Oct 31 13:02:41.975150 ip-10-0-136-168 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 13:02:42.389518 ip-10-0-136-168 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 13:02:42.389518 ip-10-0-136-168 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 13:02:42.389518 ip-10-0-136-168 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 13:02:42.389518 ip-10-0-136-168 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 13:02:42.393765 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393688    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:02:42.393765 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393764    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:02:42.393765 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393768    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393772    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393775    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393778    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393781    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393784    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393786    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393789    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393792    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393794    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393797    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393800    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393804    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393806    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393809    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393812    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393815    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393817    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393820    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393823    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:02:42.393856 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393825    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393828    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393835    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393838    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393841    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393843    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393846    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393848    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393851    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393855    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393857    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393860    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393879    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393884    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393888    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393892    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393897    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393900    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393902    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:02:42.394331 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393905    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393907    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393910    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393913    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393916    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393918    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393921    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393923    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393925    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393928    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393931    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393933    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393935    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393939    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393944    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393948    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393952    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393956    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393960    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:02:42.394776 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393963    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393966    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393969    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393972    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393974    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393977    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393980    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393982    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393984    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393987    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393990    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393992    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393995    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.393997    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394000    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394002    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394005    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394007    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394009    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394012    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:02:42.395512 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394014    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394018    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394020    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394023    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394025    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394027    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394030    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394032    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394036    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394038    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394041    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394043    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394046    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394048    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394051    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394054    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394057    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394059    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394062    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394064    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:02:42.396118 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394066    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394069    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394071    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394074    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394076    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394078    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394884    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394893    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394896    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394899    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394902    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394904    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394907    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394909    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394912    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394914    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394917    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394919    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394922    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:02:42.396592 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394925    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394927    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394959    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394996    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.394999    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395002    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395761    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395781    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395785    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395789    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395792    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395795    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395810    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395814    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395938    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395942    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395945    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395949    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395954    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:02:42.397075 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395958    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395960    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395963    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395966    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395968    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395971    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395974    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395976    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395979    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395982    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395984    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395987    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395989    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395992    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395995    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.395998    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396000    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396003    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396005    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396008    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:02:42.397533 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396011    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396014    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396016    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396019    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396027    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396030    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396033    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396036    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396038    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396041    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396043    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396048    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396052    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396056    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396060    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396063    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396066    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396068    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396071    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396074    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:02:42.398037 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396079    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396081    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396084    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396086    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396089    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396091    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396094    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396097    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396099    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396102    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396104    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396107    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396110    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396112    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396115    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396118    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396121    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396124    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396127    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:02:42.398514 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396129    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396133    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396137    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396141    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396143    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396146    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396149    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396151    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396154    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396159    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396162    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396164    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396167    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396170    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396172    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396711    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396718    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396721    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:02:42.399010 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396724    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396728    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396732    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396735    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396739    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396741    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396744    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396746    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396749    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396752    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396754    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396757    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396760    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396762    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396765    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396767    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396770    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396773    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396775    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:02:42.399436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396777    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396780    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396782    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396785    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396787    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396790    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396793    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396795    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396797    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396800    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396802    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396805    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396807    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396810    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396812    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396815    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396818    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396820    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396823    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396826    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:02:42.399896 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396828    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396831    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396833    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396836    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396838    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396841    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396843    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396845    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396848    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396850    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396853    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396856    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396858    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396861    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396879    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396883    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396886    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396888    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396891    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396893    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:02:42.400415 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396896    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396898    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396901    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396903    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396906    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396909    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396911    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396914    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396916    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396919    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396921    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396924    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396926    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396929    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396931    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396934    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396936    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396939    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396941    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396944    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:02:42.400903 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396946    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396949    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396951    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396954    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396956    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396971    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396976    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396980    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396987    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396990    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396993    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396995    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.396997    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.397000    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.397002    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.397005    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.397007    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.397010    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.397013    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:02:42.401409 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.397015    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.397017    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.397020    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.397022    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.397025    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397876    2575 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397886    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397893    2575 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397900    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397904    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397907    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397912    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397916    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397919    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397922    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397926    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397929    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397932    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397935    2575 flags.go:64] FLAG: --cgroup-root=""
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397938    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397941    2575 flags.go:64] FLAG: --client-ca-file=""
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397945    2575 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397950    2575 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 13:02:42.401847 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397955    2575 flags.go:64] FLAG: --cluster-domain=""
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397958    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397962    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397965    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397969    2575 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397973    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397977    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397980    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397983    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397987    2575 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397990    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397993    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397996    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.397999    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398003    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398006    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398009    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398012    2575 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398015    2575 flags.go:64] FLAG: --enable-server="true"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398018    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398022    2575 flags.go:64] FLAG: --event-burst="100"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398025    2575 flags.go:64] FLAG: --event-qps="50"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398028    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398031    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 13:02:42.402403 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398034    2575 flags.go:64] FLAG: --eviction-hard=""
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398038    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398042    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398045    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398048    2575 flags.go:64] FLAG: --eviction-soft=""
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398051    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398055    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398058    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398061    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398064    2575 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398067    2575 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398070    2575 flags.go:64] FLAG: --feature-gates=""
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398074    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398077    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398083    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398086    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398089    2575 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398092    2575 flags.go:64] FLAG: --help="false"
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398095    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398098    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398102    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398105    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398108    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 13:02:42.403039 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398112    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398116    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398119    2575 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398121    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398124    2575 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398127    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398131    2575 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398134    2575 flags.go:64] FLAG: --kube-reserved=""
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398137    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398140    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398143    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398146    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398149    2575 flags.go:64] FLAG: --lock-file=""
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398152    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398155    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398157    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398168    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398171    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398174    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398177    2575 flags.go:64] FLAG: --logging-format="text"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398181    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398184    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398187    2575 flags.go:64] FLAG: --manifest-url=""
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398192    2575 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398197    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 13:02:42.403596 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398201    2575 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398205    2575 flags.go:64] FLAG: --max-pods="110"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398208    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398211    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398214    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398217    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398220    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398223    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398226    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398234    2575 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398237    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398240    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398243    2575 flags.go:64] FLAG: --pod-cidr=""
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398246    2575 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398248    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398251    2575 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398255    2575 flags.go:64] FLAG: --port="10250"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398258    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398260    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-09fbf5151f39a4afe"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398264    2575 flags.go:64] FLAG: --qos-reserved=""
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398267    2575 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398270    2575 flags.go:64] FLAG: --register-node="true"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398272    2575 flags.go:64] FLAG: --register-with-taints=""
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398276    2575 flags.go:64] FLAG: --registry-burst="10"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398279    2575 flags.go:64] FLAG: --registry-qps="5"
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398282    2575 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 13:02:42.404201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398286    2575 flags.go:64] FLAG: --reserved-memory=""
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398289    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398293    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398296    2575 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398300    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398302    2575 flags.go:64] FLAG: --runonce="false"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398305    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398309    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398312    2575 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398314    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398317    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398321    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398324    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398327    2575 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398330    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398333    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398336    2575 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398339    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398343    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398345    2575 flags.go:64] FLAG: --system-cgroups=""
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398348    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398353    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398356    2575 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398359    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398364    2575 flags.go:64] FLAG: --tls-min-version=""
Oct 31 13:02:42.404874 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398366    2575 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398369    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398372    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398375    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398378    2575 flags.go:64] FLAG: --v="2"
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398382    2575 flags.go:64] FLAG: --version="false"
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398387    2575 flags.go:64] FLAG: --vmodule=""
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398390    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398394    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398528    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398531    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398535    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398538    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398540    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398543    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398545    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398548    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398552    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398556    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398561    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398564    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:02:42.405934 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398566    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398569    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398571    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398573    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398576    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398579    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398581    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398583    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398586    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398588    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398591    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398593    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398596    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398598    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398601    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398604    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398606    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398609    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398611    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398613    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:02:42.406811 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398616    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398619    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398622    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398624    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398627    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398629    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398631    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398634    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398636    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398639    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398642    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398645    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398648    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398652    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398655    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398658    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398661    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398663    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398666    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:02:42.407664 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398668    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398670    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398673    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398675    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398678    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398680    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398683    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398685    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398687    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398690    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398692    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398695    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398700    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398703    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398705    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398708    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398711    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398713    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398716    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398718    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:02:42.408525 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398721    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398723    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398725    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398730    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398732    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398735    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398737    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398740    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398742    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398744    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398747    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398749    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398752    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398754    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398756    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398759    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398761    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398763    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398766    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:02:42.409495 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398768    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:02:42.410001 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398771    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:02:42.410001 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398773    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:02:42.410001 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398776    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:02:42.410001 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398779    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:02:42.410001 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398781    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:02:42.410001 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398784    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:02:42.410001 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398786    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:02:42.410001 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398789    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:02:42.410001 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398792    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:02:42.410001 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398794    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:02:42.410001 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398797    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:02:42.410001 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398799    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:02:42.410001 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398801    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:02:42.410001 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398804    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:02:42.410001 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398806    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:02:42.410001 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.398833    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398881    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398886    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398890    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398893    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398895    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398898    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398901    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398903    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398906    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398908    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398911    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398913    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398916    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398918    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398921    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398923    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398925    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398928    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:02:42.410576 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398930    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398933    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398936    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398940    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398943    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398947    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398951    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398953    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398956    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398958    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398961    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398963    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398966    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398970    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398973    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398975    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398978    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398980    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398982    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:02:42.411371 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398985    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398987    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398990    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398992    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398995    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.398997    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399000    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399002    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399005    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399007    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399009    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399012    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399014    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399017    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399020    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399022    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399025    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399027    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399031    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399034    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:02:42.411941 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399037    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399040    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399042    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399045    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399047    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399050    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399054    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399057    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399060    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399063    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399065    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399069    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399071    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399074    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399076    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399079    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399081    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399084    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399086    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:02:42.412436 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399089    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399091    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399093    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399096    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399098    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399101    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399103    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399106    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399108    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399110    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399113    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399117    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399119    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399122    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399129    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399133    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399136    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399140    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399142    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:02:42.412952 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399146    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399148    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399151    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399153    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399156    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399158    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399161    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399163    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399166    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399168    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.399171    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.407441    2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.407458    2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.407473    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.407481    2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407566    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407575    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407581    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407585    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407590    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:02:42.413539 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407595    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407599    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407604    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407608    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407613    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407618    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407621    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407627    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407632    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407636    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407640    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407645    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407649    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407654    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407660    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407665    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407669    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407674    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407678    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:02:42.414122 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407682    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407687    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407691    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407696    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407700    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407706    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407711    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407715    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407719    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407723    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407727    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407731    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407735    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407740    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407744    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407749    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407753    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407757    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407761    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407765    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:02:42.414632 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407770    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407775    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407780    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407784    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407788    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407792    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407796    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407800    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407804    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407808    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407813    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407817    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407821    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407825    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407832    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407837    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407842    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407847    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:02:42.415127 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407851    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407855    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407860    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407885    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407890    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407895    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407900    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407904    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407908    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407912    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407916    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407921    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407925    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407929    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407933    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407937    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407943    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407947    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407951    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407955    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:02:42.415584 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407960    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407964    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407968    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407972    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407976    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407980    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407985    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407989    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407993    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.407997    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408001    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408006    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408010    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408014    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408018    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408022    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408026    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408031    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408034    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408039    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:02:42.416169 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408043    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:02:42.416630 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408048    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:02:42.416630 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408052    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:02:42.416630 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408056    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:02:42.416630 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.408097    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 13:02:42.416630 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408317    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:02:42.416630 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408326    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:02:42.416630 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408331    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:02:42.416630 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408335    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:02:42.416630 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408340    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:02:42.416630 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408344    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:02:42.416630 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408348    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:02:42.416630 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408352    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:02:42.416630 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408357    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:02:42.416630 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408361    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:02:42.416630 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408365    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:02:42.416630 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408369    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408373    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408378    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408382    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408386    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408390    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408394    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408401    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408407    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408412    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408416    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408421    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408425    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408429    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408433    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408437    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408441    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408445    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408449    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:02:42.417044 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408455    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408460    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408464    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408469    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408473    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408477    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408481    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408486    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408490    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408494    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408499    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408503    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408507    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408511    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408515    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408520    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408524    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408527    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408531    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408535    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:02:42.417793 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408539    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408543    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408547    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408551    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408555    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408559    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408563    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408567    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408571    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408575    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408579    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408583    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408587    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408591    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408595    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408599    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408603    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408607    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408611    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408615    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:02:42.418603 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408620    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408624    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408628    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408639    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408643    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408647    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408654    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408660    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408664    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408668    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408672    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408676    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408680    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408685    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408688    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408693    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408697    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408701    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408705    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408709    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:02:42.419441 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408713    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:02:42.419967 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408717    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:02:42.419967 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408721    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:02:42.419967 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408725    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:02:42.419967 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408729    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:02:42.419967 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408734    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:02:42.419967 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408738    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:02:42.419967 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408742    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:02:42.419967 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408746    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:02:42.419967 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408750    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:02:42.419967 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408754    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:02:42.419967 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408758    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:02:42.419967 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408762    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:02:42.419967 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408767    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:02:42.419967 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408771    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:02:42.419967 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408775    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:02:42.419967 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.408814    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408887    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408894    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408901    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408905    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408909    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408912    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408916    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408921    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408925    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408929    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408933    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408936    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408940    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408944    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408947    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408951    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408955    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408958    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408962    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:02:42.420369 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408967    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408971    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408975    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408979    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408983    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408988    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408992    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.408996    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409000    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409003    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409009    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409013    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409017    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409021    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409025    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409031    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409035    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409040    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409044    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:02:42.420886 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409047    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409052    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409056    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409063    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409069    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409074    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409079    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409085    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409089    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409093    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409097    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409101    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409105    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409109    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409114    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409121    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409126    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409130    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409134    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:02:42.421364 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409139    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409143    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409147    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409151    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409155    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409160    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409165    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409169    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409173    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409179    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409184    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409188    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409192    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409197    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409201    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409205    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409210    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409214    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409218    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:02:42.421831 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409222    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409227    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409231    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409235    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409239    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409243    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409247    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409252    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409256    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409260    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409264    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409270    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409274    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409278    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409282    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409286    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409290    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409294    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409298    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409303    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:02:42.423096 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409307    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:02:42.423589 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409311    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:02:42.423589 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409318    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:02:42.423589 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409323    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:02:42.423589 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409328    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:02:42.423589 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409333    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:02:42.423589 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409337    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:02:42.423589 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409342    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:02:42.423589 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409346    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:02:42.423589 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:42.409350    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:02:42.423589 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.409430    2575 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 13:02:42.423589 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.414042    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 13:02:42.423589 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.415138    2575 server.go:1014] "Starting client certificate rotation"
Oct 31 13:02:42.423589 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.415238    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 13:02:42.423589 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.415274    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 13:02:42.423589 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.417759    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 13:02:42.423589 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.419494    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 13:02:42.436565 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.436548    2575 log.go:25] "Validated CRI v1 runtime API"
Oct 31 13:02:42.442918 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.442903    2575 log.go:25] "Validated CRI v1 image API"
Oct 31 13:02:42.443770 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.443749    2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 13:02:42.449434 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.449411    2575 fs.go:135] Filesystem UUIDs: map[157a36a1-2b0e-4f40-bb36-d5a3a8aa8c83:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 eb16f3e7-25bd-4f05-91d4-89ab7201b35a:/dev/nvme0n1p3]
Oct 31 13:02:42.449506 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.449432    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 13:02:42.454491 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.454375    2575 manager.go:217] Machine: {Timestamp:2026-10-31 13:02:42.453890891 +0000 UTC m=+0.364366120 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110340 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aec95a5185cd44cb52f63895644d2 SystemUUID:ec2aec95-a518-5cd4-4cb5-2f63895644d2 BootID:32d5510d-fce8-4e49-9640-e81e3b8edf80 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f7:4a:7f:67 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f7:4a:7f:67 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:f1:80:3a:1a:77 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 13:02:42.454491 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.454478    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 13:02:42.454650 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.454629    2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 13:02:42.456555 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.456530    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 13:02:42.458131 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.458104    2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 13:02:42.458300 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.458136    2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-168.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 13:02:42.458386 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.458304    2575 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 13:02:42.458386 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.458318    2575 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 13:02:42.458386 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.458350    2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 13:02:42.459170 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.459158    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 13:02:42.460414 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.460402    2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 13:02:42.464790 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.464776    2575 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 13:02:42.464942 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.464931    2575 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 13:02:42.465009 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.464971    2575 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 13:02:42.465009 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.464993    2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 13:02:42.465009 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.465007    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 13:02:42.465171 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.465020    2575 kubelet.go:405] "Adding apiserver pod source"
Oct 31 13:02:42.465171 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.465031    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 13:02:42.465394 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.465379    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 13:02:42.467846 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.467831    2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 13:02:42.474380 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.474366    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 13:02:42.475432 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.475420    2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 13:02:42.475466 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.475451    2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 13:02:42.475606 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.475596    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 13:02:42.475634 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.475612    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 13:02:42.475634 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.475619    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 13:02:42.475634 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.475625    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 13:02:42.475634 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.475630    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 13:02:42.475634 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.475635    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 13:02:42.476382 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.476369    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 13:02:42.476382 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.476381    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 13:02:42.476471 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.476390    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 13:02:42.476471 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.476396    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 13:02:42.476471 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.476414    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 13:02:42.476471 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.476424    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 13:02:42.478100 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.478090    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 13:02:42.478155 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.478104    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 13:02:42.480810 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.480414    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tw49p"
Oct 31 13:02:42.482912 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.482892    2575 server.go:1263] "Started kubelet"
Oct 31 13:02:42.483070 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.483035    2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 13:02:42.483177 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.483077    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 13:02:42.483177 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.483130    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 13:02:42.483755 ip-10-0-136-168 systemd[1]: Started Kubernetes Kubelet.
Oct 31 13:02:42.484789 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.484731    2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 13:02:42.485795 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.485781    2575 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 13:02:42.488537 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:42.488514    2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 13:02:42.491766 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.491744    2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 13:02:42.491766 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.491752    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 13:02:42.491910 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:42.491860    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-168.us-east-2.compute.internal\" not found"
Oct 31 13:02:42.492501 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.492484    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 13:02:42.492501 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.492484    2575 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 13:02:42.492653 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.492510    2575 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 13:02:42.492653 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.492586    2575 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 13:02:42.492653 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.492595    2575 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 13:02:42.493048 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:42.493021    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-168.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 13:02:42.493134 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.493118    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-168.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 13:02:42.493196 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:42.493143    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 13:02:42.494419 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.494120    2575 factory.go:55] Registering systemd factory
Oct 31 13:02:42.494419 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.494138    2575 factory.go:223] Registration of the systemd container factory successfully
Oct 31 13:02:42.494550 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.494522    2575 factory.go:153] Registering CRI-O factory
Oct 31 13:02:42.494550 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.494536    2575 factory.go:223] Registration of the crio container factory successfully
Oct 31 13:02:42.494643 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.494582    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 13:02:42.494643 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.494601    2575 factory.go:103] Registering Raw factory
Oct 31 13:02:42.494643 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.494616    2575 manager.go:1199] Started watching for new ooms in manager
Oct 31 13:02:42.494895 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:42.492963    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-168.us-east-2.compute.internal.18e39ee038967089  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-168.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-168.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 13:02:42.482835593 +0000 UTC m=+0.393310802,LastTimestamp:2026-10-31 13:02:42.482835593 +0000 UTC m=+0.393310802,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-168.us-east-2.compute.internal,}"
Oct 31 13:02:42.495661 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.495638    2575 manager.go:319] Starting recovery of all containers
Oct 31 13:02:42.496855 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.496788    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tw49p"
Oct 31 13:02:42.499414 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:42.499384    2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-168.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 13:02:42.509561 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.509543    2575 manager.go:324] Recovery completed
Oct 31 13:02:42.511420 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.511403    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:02:42.513366 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.513352    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:02:42.515133 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.515114    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:02:42.515219 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.515148    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:02:42.515219 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.515163    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:02:42.515736 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.515720    2575 cpu_manager.go:226] "Starting" policy="none"
Oct 31 13:02:42.515736 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.515736    2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 13:02:42.515846 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.515743    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:02:42.515846 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.515751    2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 13:02:42.516483 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.516468    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:02:42.516549 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.516493    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:02:42.516549 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.516503    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:02:42.517833 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.517821    2575 policy_none.go:50] "Start"
Oct 31 13:02:42.517895 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.517837    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 13:02:42.517895 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.517847    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 13:02:42.519894 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.519883    2575 policy_none.go:44] "Start"
Oct 31 13:02:42.575389 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.557445    2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 13:02:42.575389 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.561827    2575 manager.go:341] "Starting Device Plugin manager"
Oct 31 13:02:42.575389 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:42.561886    2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 13:02:42.575389 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.561899    2575 server.go:87] "Starting device plugin registration server"
Oct 31 13:02:42.575389 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.562146    2575 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 13:02:42.575389 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.562158    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 13:02:42.575389 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.562260    2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 13:02:42.575389 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.562382    2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 13:02:42.575389 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.562391    2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 13:02:42.575389 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:42.562860    2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 13:02:42.575389 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:42.562921    2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-168.us-east-2.compute.internal\" not found"
Oct 31 13:02:42.575389 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.568095    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 13:02:42.575389 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.569297    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 13:02:42.575389 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.569313    2575 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 13:02:42.575389 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.569332    2575 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 13:02:42.575389 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:42.569359    2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 13:02:42.575389 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.573358    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:02:42.662830 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.662767    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:02:42.663587 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.663571    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:02:42.663699 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.663600    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:02:42.663699 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.663610    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:02:42.663699 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.663637    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:42.670164 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.670146    2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal"]
Oct 31 13:02:42.670239 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.670200    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:02:42.671921 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.671907    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:02:42.671991 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.671936    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:02:42.671991 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.671947    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:02:42.672694 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.672676    2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:42.672771 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:42.672697    2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-168.us-east-2.compute.internal\": node \"ip-10-0-136-168.us-east-2.compute.internal\" not found"
Oct 31 13:02:42.674029 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.674012    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:02:42.674159 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.674145    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:42.674199 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.674177    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:02:42.675058 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.675039    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:02:42.675137 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.675072    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:02:42.675137 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.675042    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:02:42.675137 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.675090    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:02:42.675137 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.675100    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:02:42.675137 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.675114    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:02:42.677185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.677165    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:42.677279 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.677193    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:02:42.677925 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.677911    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:02:42.678002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.677937    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:02:42.678002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.677950    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:02:42.694819 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.694802    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62681df534845faea6eef1fcf975d228-config\") pod \"kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"62681df534845faea6eef1fcf975d228\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:42.694894 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.694828    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91e8205bd2503f1072bc57fd6d59d36b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"91e8205bd2503f1072bc57fd6d59d36b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:42.694894 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.694843    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91e8205bd2503f1072bc57fd6d59d36b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"91e8205bd2503f1072bc57fd6d59d36b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:42.697389 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:42.697374    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-168.us-east-2.compute.internal\" not found"
Oct 31 13:02:42.709335 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:42.709319    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-168.us-east-2.compute.internal\" not found" node="ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:42.711971 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:42.711948    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-168.us-east-2.compute.internal\" not found" node="ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:42.795324 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.795301    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62681df534845faea6eef1fcf975d228-config\") pod \"kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"62681df534845faea6eef1fcf975d228\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:42.795428 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.795325    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91e8205bd2503f1072bc57fd6d59d36b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"91e8205bd2503f1072bc57fd6d59d36b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:42.795428 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.795343    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91e8205bd2503f1072bc57fd6d59d36b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"91e8205bd2503f1072bc57fd6d59d36b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:42.795428 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.795369    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91e8205bd2503f1072bc57fd6d59d36b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"91e8205bd2503f1072bc57fd6d59d36b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:42.795428 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.795388    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62681df534845faea6eef1fcf975d228-config\") pod \"kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"62681df534845faea6eef1fcf975d228\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:42.795428 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:42.795415    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91e8205bd2503f1072bc57fd6d59d36b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"91e8205bd2503f1072bc57fd6d59d36b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:42.798393 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:42.798379    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-168.us-east-2.compute.internal\" not found"
Oct 31 13:02:42.898961 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:42.898938    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-168.us-east-2.compute.internal\" not found"
Oct 31 13:02:42.999750 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:42.999727    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-168.us-east-2.compute.internal\" not found"
Oct 31 13:02:43.014911 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:43.014892    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:43.017035 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:43.017020    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:43.100612 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:43.100584    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-168.us-east-2.compute.internal\" not found"
Oct 31 13:02:43.201059 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:43.201031    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-168.us-east-2.compute.internal\" not found"
Oct 31 13:02:43.301615 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:43.301565    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-168.us-east-2.compute.internal\" not found"
Oct 31 13:02:43.402085 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:43.402060    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-168.us-east-2.compute.internal\" not found"
Oct 31 13:02:43.415204 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:43.415183    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 13:02:43.415331 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:43.415314    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 13:02:43.415383 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:43.415346    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 13:02:43.502357 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:43.502326    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-168.us-east-2.compute.internal\" not found"
Oct 31 13:02:43.503448 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:43.503400    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:57:42 +0000 UTC" deadline="2028-06-06 10:41:24.677513846 +0000 UTC"
Oct 31 13:02:43.503492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:43.503448    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14013h38m41.174068506s"
Oct 31 13:02:43.603163 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:43.603136    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-168.us-east-2.compute.internal\" not found"
Oct 31 13:02:43.606010 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:43.605989    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:02:43.685169 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:43.685138    2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:02:43.692178 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:43.692161    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:43.708535 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:43.708518    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 13:02:43.710627 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:43.710611    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:02:43.715750 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:43.715707    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e8205bd2503f1072bc57fd6d59d36b.slice/crio-c82bdb7624c5c7e1724272d83a313273acd800ceae755fe5179ccfab6d9d1ff8 WatchSource:0}: Error finding container c82bdb7624c5c7e1724272d83a313273acd800ceae755fe5179ccfab6d9d1ff8: Status 404 returned error can't find the container with id c82bdb7624c5c7e1724272d83a313273acd800ceae755fe5179ccfab6d9d1ff8
Oct 31 13:02:43.716107 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:43.716089    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62681df534845faea6eef1fcf975d228.slice/crio-d0007b66a1deb53d8b3459e5c7a8d692ea642205e62878bc20b1f25b1955d151 WatchSource:0}: Error finding container d0007b66a1deb53d8b3459e5c7a8d692ea642205e62878bc20b1f25b1955d151: Status 404 returned error can't find the container with id d0007b66a1deb53d8b3459e5c7a8d692ea642205e62878bc20b1f25b1955d151
Oct 31 13:02:43.721345 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:43.721324    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 13:02:43.721451 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:43.721408    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:02:44.466601 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.466570    2575 apiserver.go:52] "Watching apiserver"
Oct 31 13:02:44.477452 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.477414    2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 13:02:44.477837 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.477809    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-s49pl","openshift-network-diagnostics/network-check-target-xpskx","openshift-multus/multus-dxmqp","openshift-network-operator/iptables-alerter-7bpj5","kube-system/global-pull-secret-syncer-6tcxq","openshift-dns/node-resolver-z65rw","openshift-ovn-kubernetes/ovnkube-node-vl2d9","kube-system/konnectivity-agent-ff7kw","kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal","openshift-image-registry/node-ca-nmwhw","openshift-multus/multus-additional-cni-plugins-x6wvd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9","openshift-cluster-node-tuning-operator/tuned-rzs7m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"]
Oct 31 13:02:44.480957 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.480933    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.483505 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.483479    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.486068 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.486043    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.487001 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.486970    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 13:02:44.487126 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.487073    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 13:02:44.487283 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.487268    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 13:02:44.487327 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.487320    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 13:02:44.488122 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.488107    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r2wwf\""
Oct 31 13:02:44.488201 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.488143    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 13:02:44.488277 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.488266    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:02:44.489146 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.489053    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:02:44.489146 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.489105    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 13:02:44.489353 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.489173    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s8mtw\""
Oct 31 13:02:44.489353 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.489326    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 13:02:44.492162 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.490571    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:44.492162 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:44.490658    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tcxq" podUID="d7a89a36-092a-4d38-8a50-60619cc50832"
Oct 31 13:02:44.492162 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.491106    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 13:02:44.492162 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.491394    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 13:02:44.492162 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.491423    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 13:02:44.492162 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.491487    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 13:02:44.492162 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.491520    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-s9d6g\""
Oct 31 13:02:44.492794 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.492775    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 13:02:44.493668 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.493651    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:02:44.493772 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.493756    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 13:02:44.493853 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.493836    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkxw2\""
Oct 31 13:02:44.493901 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.493873    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 13:02:44.494291 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.494273    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:02:44.499038 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.499015    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ff7kw"
Oct 31 13:02:44.500593 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.499911    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:02:44.500593 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:44.500054    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s49pl" podUID="0edfba66-6a3a-4be8-b420-227d0443ddea"
Oct 31 13:02:44.500593 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.500310    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-65v8h\""
Oct 31 13:02:44.502880 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.502855    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78mh\" (UniqueName: \"kubernetes.io/projected/f24ca8d4-b321-4579-aff7-e1d0177f50bc-kube-api-access-m78mh\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.503046 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.502898    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-cni-netd\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.503046 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.502925    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-systemd\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.503046 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503006    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-tmp\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.503046 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503031    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-system-cni-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.503238 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503056    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-netns\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.503238 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503080    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-systemd-units\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.503238 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503209    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-etc-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.503391 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503235    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysctl-d\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.503391 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503303    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:02:44.503493 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:44.503396    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpskx" podUID="12e357d7-757d-4f89-837e-6283d4f87e11"
Oct 31 13:02:44.503493 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503298    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-ovnkube-config\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.503493 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503453    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-socket-dir-parent\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.503493 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-tmp-dir\") pod \"node-resolver-z65rw\" (UID: \"62aae01d-fd1d-43c3-9ad7-965bc6d595e2\") " pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:02:44.503698 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503510    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysctl-conf\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.503698 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503536    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.503698 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503562    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-tuned\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.503698 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503589    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-cni-bin\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.503698 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503613    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7a89a36-092a-4d38-8a50-60619cc50832-kubelet-config\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:44.503698 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503639    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-var-lib-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.503698 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503664    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-log-socket\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.503698 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503690    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-kubelet\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.504130 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503713    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-cni-bin\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.504130 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503737    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-run\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.504130 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503759    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-hosts-file\") pod \"node-resolver-z65rw\" (UID: \"62aae01d-fd1d-43c3-9ad7-965bc6d595e2\") " pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:02:44.504130 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503788    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-node-log\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.504130 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503812    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-run-ovn-kubernetes\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.504130 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503837    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-cni-multus\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.504130 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503861    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-conf-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.504130 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503898    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-daemon-config\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.504130 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503950    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-slash\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.504130 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503974    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-systemd\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.504130 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.503996    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-var-lib-kubelet\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.504130 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504023    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xl4l\" (UniqueName: \"kubernetes.io/projected/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-kube-api-access-5xl4l\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.504130 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504048    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-iptables-alerter-script\") pod \"iptables-alerter-7bpj5\" (UID: \"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d\") " pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:02:44.504130 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504075    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5hcb\" (UniqueName: \"kubernetes.io/projected/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-kube-api-access-s5hcb\") pod \"node-resolver-z65rw\" (UID: \"62aae01d-fd1d-43c3-9ad7-965bc6d595e2\") " pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:02:44.504130 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504099    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-host\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.504130 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504123    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f24ca8d4-b321-4579-aff7-e1d0177f50bc-cni-binary-copy\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.505002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504147    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-k8s-cni-cncf-io\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.505002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504172    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94j67\" (UniqueName: \"kubernetes.io/projected/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-kube-api-access-94j67\") pod \"iptables-alerter-7bpj5\" (UID: \"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d\") " pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:02:44.505002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504200    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:44.505002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504229    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-ovn\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.505002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504253    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-sys\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.505002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504281    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.505002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504307    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/103c4c86-a9f6-4a59-a615-940717e39dc9-ovn-node-metrics-cert\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.505002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504332    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-modprobe-d\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.505002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504361    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-kubernetes\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.505002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504385    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-cni-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.505002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504414    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-etc-kubernetes\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.505002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-ovnkube-script-lib\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.505002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504463    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7cnz\" (UniqueName: \"kubernetes.io/projected/103c4c86-a9f6-4a59-a615-940717e39dc9-kube-api-access-f7cnz\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.505002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504485    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysconfig\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.505002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504510    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-kubelet\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.505002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504533    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-lib-modules\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.505919 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504559    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-hostroot\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.505919 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504584    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-host-slash\") pod \"iptables-alerter-7bpj5\" (UID: \"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d\") " pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:02:44.505919 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504611    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-env-overrides\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.505919 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504647    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-multus-certs\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.505919 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504672    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7a89a36-092a-4d38-8a50-60619cc50832-dbus\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:44.505919 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504702    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-cnibin\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.505919 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.504729    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-run-netns\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.505919 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.505811    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:02:44.508260 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.508242    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.510679 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.510658    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.516611 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.516590    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 13:02:44.532806 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.532778    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-c6w9z"
Oct 31 13:02:44.544762 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.544741    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-c6w9z"
Oct 31 13:02:44.574901 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.574848    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal" event={"ID":"62681df534845faea6eef1fcf975d228","Type":"ContainerStarted","Data":"d0007b66a1deb53d8b3459e5c7a8d692ea642205e62878bc20b1f25b1955d151"}
Oct 31 13:02:44.577419 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.577396    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal" event={"ID":"91e8205bd2503f1072bc57fd6d59d36b","Type":"ContainerStarted","Data":"c82bdb7624c5c7e1724272d83a313273acd800ceae755fe5179ccfab6d9d1ff8"}
Oct 31 13:02:44.594157 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.594099    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 13:02:44.595924 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.595561    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 13:02:44.595924 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.595879    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 13:02:44.595924 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.595889    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 13:02:44.596118 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.595972    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 13:02:44.596118 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.595975    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 13:02:44.596894 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.596677    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 13:02:44.596894 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.596730    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 13:02:44.596894 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.596749    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lxhpt\""
Oct 31 13:02:44.596894 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.596776    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 13:02:44.596894 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.596787    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 13:02:44.596894 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.596797    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 13:02:44.596894 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.596865    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 13:02:44.597317 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.596963    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-glfvs\""
Oct 31 13:02:44.597317 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.597057    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 13:02:44.599209 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.599189    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-h72qs\""
Oct 31 13:02:44.599827 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.599807    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tmkp5\""
Oct 31 13:02:44.605969 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.605942    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-log-socket\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.606060 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.605982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-kubelet\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.606060 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606014    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-cni-bin\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.606060 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606038    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-run\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.606269 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606063    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-hosts-file\") pod \"node-resolver-z65rw\" (UID: \"62aae01d-fd1d-43c3-9ad7-965bc6d595e2\") " pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:02:44.606269 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606049    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-log-socket\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.606269 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606095    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-kubelet\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.606269 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606088    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-node-log\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.606269 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606135    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-node-log\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.606269 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606148    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-run-ovn-kubernetes\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.606269 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606197    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-hosts-file\") pod \"node-resolver-z65rw\" (UID: \"62aae01d-fd1d-43c3-9ad7-965bc6d595e2\") " pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:02:44.606269 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606145    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-cni-bin\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.606269 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606195    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-run-ovn-kubernetes\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.606269 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606176    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-cni-multus\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.606269 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606249    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-run\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.606269 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-conf-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.606877 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606286    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-daemon-config\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.606877 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606324    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-cni-multus\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.606877 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606331    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb2z7\" (UniqueName: \"kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7\") pod \"network-check-target-xpskx\" (UID: \"12e357d7-757d-4f89-837e-6283d4f87e11\") " pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:02:44.606877 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606364    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-conf-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.606877 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606388    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-slash\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.606877 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606415    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-systemd\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.606877 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606432    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-var-lib-kubelet\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.606877 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606448    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xl4l\" (UniqueName: \"kubernetes.io/projected/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-kube-api-access-5xl4l\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.606877 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606469    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-iptables-alerter-script\") pod \"iptables-alerter-7bpj5\" (UID: \"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d\") " pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:02:44.606877 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606495    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5hcb\" (UniqueName: \"kubernetes.io/projected/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-kube-api-access-s5hcb\") pod \"node-resolver-z65rw\" (UID: \"62aae01d-fd1d-43c3-9ad7-965bc6d595e2\") " pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:02:44.606877 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606504    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-systemd\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.606877 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606521    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-var-lib-kubelet\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.606877 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606525    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhrzc\" (UniqueName: \"kubernetes.io/projected/0edfba66-6a3a-4be8-b420-227d0443ddea-kube-api-access-zhrzc\") pod \"network-metrics-daemon-s49pl\" (UID: \"0edfba66-6a3a-4be8-b420-227d0443ddea\") " pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:02:44.606877 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606549    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-slash\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.606877 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606552    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-socket-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.607598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606927    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-host\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.607598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606934    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-daemon-config\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.607598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606960    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f24ca8d4-b321-4579-aff7-e1d0177f50bc-cni-binary-copy\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.607598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.606984    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-k8s-cni-cncf-io\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.607598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607010    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94j67\" (UniqueName: \"kubernetes.io/projected/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-kube-api-access-94j67\") pod \"iptables-alerter-7bpj5\" (UID: \"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d\") " pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:02:44.607598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607037    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:44.607598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607045    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-k8s-cni-cncf-io\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.607598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607012    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-host\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.607598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607066    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-cnibin\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.607598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607092    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-ovn\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.607598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607114    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-sys\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.607598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607138    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82sb9\" (UniqueName: \"kubernetes.io/projected/2f97d160-4ad4-4546-9310-4a20d1309648-kube-api-access-82sb9\") pod \"node-ca-nmwhw\" (UID: \"2f97d160-4ad4-4546-9310-4a20d1309648\") " pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:02:44.607598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607141    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-iptables-alerter-script\") pod \"iptables-alerter-7bpj5\" (UID: \"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d\") " pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:02:44.607598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607165    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.607598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607190    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/103c4c86-a9f6-4a59-a615-940717e39dc9-ovn-node-metrics-cert\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.607598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607213    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-modprobe-d\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.607598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607219    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-ovn\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.608323 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607237    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-kubernetes\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.608323 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607262    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-cni-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.608323 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607286    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-etc-kubernetes\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.608323 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:44.607308    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:44.608323 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:44.607401    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret podName:d7a89a36-092a-4d38-8a50-60619cc50832 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:45.107355738 +0000 UTC m=+3.017830952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret") pod "global-pull-secret-syncer-6tcxq" (UID: "d7a89a36-092a-4d38-8a50-60619cc50832") : object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:44.608323 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607431    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f24ca8d4-b321-4579-aff7-e1d0177f50bc-cni-binary-copy\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.608323 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607310    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-ovnkube-script-lib\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.608323 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607482    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7cnz\" (UniqueName: \"kubernetes.io/projected/103c4c86-a9f6-4a59-a615-940717e39dc9-kube-api-access-f7cnz\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.608323 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607511    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysconfig\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.608323 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607536    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-kubelet\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.608323 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607565    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs\") pod \"network-metrics-daemon-s49pl\" (UID: \"0edfba66-6a3a-4be8-b420-227d0443ddea\") " pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:02:44.608323 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607628    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-etc-selinux\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.608323 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607638    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysconfig\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.608323 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607657    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-lib-modules\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.608323 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607683    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-hostroot\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.608323 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607708    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-host-slash\") pod \"iptables-alerter-7bpj5\" (UID: \"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d\") " pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:02:44.608323 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607737    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-env-overrides\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607793    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-kubelet\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607824    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-multus-certs\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607852    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7a89a36-092a-4d38-8a50-60619cc50832-dbus\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607869    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-ovnkube-script-lib\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607877    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607704    2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607924    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmbrx\" (UniqueName: \"kubernetes.io/projected/8d617e36-a72a-4a8c-bc9c-85500f109e42-kube-api-access-bmbrx\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.607963    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-modprobe-d\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608091    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7a89a36-092a-4d38-8a50-60619cc50832-dbus\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608092    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-cni-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608123    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-etc-kubernetes\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608150    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-kubernetes\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608159    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-env-overrides\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608168    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-sys\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608211    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-multus-certs\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608213    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1b19e39-5b5d-46f7-b005-a1e5165fe135-agent-certs\") pod \"konnectivity-agent-ff7kw\" (UID: \"c1b19e39-5b5d-46f7-b005-a1e5165fe135\") " pod="kube-system/konnectivity-agent-ff7kw"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608268    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-hostroot\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.609185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608270    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.609754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608274    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-lib-modules\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.609754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608301    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-cnibin\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.609754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608327    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-host-slash\") pod \"iptables-alerter-7bpj5\" (UID: \"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d\") " pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:02:44.609754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608375    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-registration-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.609754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608416    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-cnibin\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.609754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608439    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-run-netns\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.609754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608482    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78mh\" (UniqueName: \"kubernetes.io/projected/f24ca8d4-b321-4579-aff7-e1d0177f50bc-kube-api-access-m78mh\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.609754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608509    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.609754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608553    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-run-netns\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.609754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608569    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.609754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1b19e39-5b5d-46f7-b005-a1e5165fe135-konnectivity-ca\") pod \"konnectivity-agent-ff7kw\" (UID: \"c1b19e39-5b5d-46f7-b005-a1e5165fe135\") " pod="kube-system/konnectivity-agent-ff7kw"
Oct 31 13:02:44.609754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608719    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.609754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608742    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-device-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.609754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608805    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-cni-netd\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.609754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608835    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-systemd\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.609754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608863    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-tmp\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.609754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608887    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-system-cni-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.610492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.608972    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-systemd\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.610492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609014    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-cni-netd\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.610492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609044    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-netns\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.610492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609073    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-system-cni-dir\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.610492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609113    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdl5n\" (UniqueName: \"kubernetes.io/projected/82689b74-1468-4087-8123-b361839b4124-kube-api-access-tdl5n\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.610492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609156    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-systemd-units\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.610492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609181    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-etc-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.610492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609205    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f97d160-4ad4-4546-9310-4a20d1309648-serviceca\") pod \"node-ca-nmwhw\" (UID: \"2f97d160-4ad4-4546-9310-4a20d1309648\") " pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:02:44.610492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609231    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysctl-d\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.610492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609256    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-ovnkube-config\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.610492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609283    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-socket-dir-parent\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.610492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609344    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-tmp-dir\") pod \"node-resolver-z65rw\" (UID: \"62aae01d-fd1d-43c3-9ad7-965bc6d595e2\") " pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:02:44.610492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609372    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f97d160-4ad4-4546-9310-4a20d1309648-host\") pod \"node-ca-nmwhw\" (UID: \"2f97d160-4ad4-4546-9310-4a20d1309648\") " pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:02:44.610492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609399    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-cni-binary-copy\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.610492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609425    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysctl-conf\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.610492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609451    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.610492 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609476    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-tuned\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.611335 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609506    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-cni-bin\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.611335 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609535    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7a89a36-092a-4d38-8a50-60619cc50832-kubelet-config\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:44.611335 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609559    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-var-lib-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.611335 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609587    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-sys-fs\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.611335 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.609680    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-socket-dir-parent\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.611335 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.610030    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-systemd-units\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.611335 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.610032    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.611335 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.610094    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-system-cni-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.611335 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.610133    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-netns\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.611335 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.610181    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-tmp-dir\") pod \"node-resolver-z65rw\" (UID: \"62aae01d-fd1d-43c3-9ad7-965bc6d595e2\") " pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:02:44.611335 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.610193    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysctl-conf\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.611335 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.610196    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-cni-bin\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.611335 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.610234    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-etc-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.611335 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.610303    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysctl-d\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.611335 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.610348    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-var-lib-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.611335 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.610393    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7a89a36-092a-4d38-8a50-60619cc50832-kubelet-config\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:44.611335 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.610741    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-ovnkube-config\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.612487 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.612464    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-tmp\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.614560 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.614490    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/103c4c86-a9f6-4a59-a615-940717e39dc9-ovn-node-metrics-cert\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.616662 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.616639    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-tuned\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.623870 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.623846    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xl4l\" (UniqueName: \"kubernetes.io/projected/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-kube-api-access-5xl4l\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.624160 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.624117    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5hcb\" (UniqueName: \"kubernetes.io/projected/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-kube-api-access-s5hcb\") pod \"node-resolver-z65rw\" (UID: \"62aae01d-fd1d-43c3-9ad7-965bc6d595e2\") " pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:02:44.624287 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.624271    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78mh\" (UniqueName: \"kubernetes.io/projected/f24ca8d4-b321-4579-aff7-e1d0177f50bc-kube-api-access-m78mh\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.624402 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.624386    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94j67\" (UniqueName: \"kubernetes.io/projected/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-kube-api-access-94j67\") pod \"iptables-alerter-7bpj5\" (UID: \"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d\") " pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:02:44.624693 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.624674    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7cnz\" (UniqueName: \"kubernetes.io/projected/103c4c86-a9f6-4a59-a615-940717e39dc9-kube-api-access-f7cnz\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.651321 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.651299    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:02:44.710509 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.710472    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f97d160-4ad4-4546-9310-4a20d1309648-host\") pod \"node-ca-nmwhw\" (UID: \"2f97d160-4ad4-4546-9310-4a20d1309648\") " pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:02:44.710639 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.710516    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-cni-binary-copy\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.710639 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.710547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-sys-fs\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.710639 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.710570    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb2z7\" (UniqueName: \"kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7\") pod \"network-check-target-xpskx\" (UID: \"12e357d7-757d-4f89-837e-6283d4f87e11\") " pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:02:44.710639 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.710599    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhrzc\" (UniqueName: \"kubernetes.io/projected/0edfba66-6a3a-4be8-b420-227d0443ddea-kube-api-access-zhrzc\") pod \"network-metrics-daemon-s49pl\" (UID: \"0edfba66-6a3a-4be8-b420-227d0443ddea\") " pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:02:44.710639 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.710601    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f97d160-4ad4-4546-9310-4a20d1309648-host\") pod \"node-ca-nmwhw\" (UID: \"2f97d160-4ad4-4546-9310-4a20d1309648\") " pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:02:44.710639 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.710619    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-socket-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.710978 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.710639    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-sys-fs\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.710978 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.710683    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-cnibin\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.710978 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.710718    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82sb9\" (UniqueName: \"kubernetes.io/projected/2f97d160-4ad4-4546-9310-4a20d1309648-kube-api-access-82sb9\") pod \"node-ca-nmwhw\" (UID: \"2f97d160-4ad4-4546-9310-4a20d1309648\") " pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:02:44.710978 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.710752    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs\") pod \"network-metrics-daemon-s49pl\" (UID: \"0edfba66-6a3a-4be8-b420-227d0443ddea\") " pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:02:44.710978 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.710777    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-socket-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.710978 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.710880    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-cnibin\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.710978 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.710889    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-etc-selinux\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.710978 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.710778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-etc-selinux\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.710978 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:44.710883    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:44.710978 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.710967    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.711295 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.710996    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmbrx\" (UniqueName: \"kubernetes.io/projected/8d617e36-a72a-4a8c-bc9c-85500f109e42-kube-api-access-bmbrx\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.711295 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:44.711027    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs podName:0edfba66-6a3a-4be8-b420-227d0443ddea nodeName:}" failed. No retries permitted until 2026-10-31 13:02:45.211008632 +0000 UTC m=+3.121483846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs") pod "network-metrics-daemon-s49pl" (UID: "0edfba66-6a3a-4be8-b420-227d0443ddea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:44.711295 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711048    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1b19e39-5b5d-46f7-b005-a1e5165fe135-agent-certs\") pod \"konnectivity-agent-ff7kw\" (UID: \"c1b19e39-5b5d-46f7-b005-a1e5165fe135\") " pod="kube-system/konnectivity-agent-ff7kw"
Oct 31 13:02:44.711295 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711080    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-registration-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.711295 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.711295 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711141    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.711295 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711170    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1b19e39-5b5d-46f7-b005-a1e5165fe135-konnectivity-ca\") pod \"konnectivity-agent-ff7kw\" (UID: \"c1b19e39-5b5d-46f7-b005-a1e5165fe135\") " pod="kube-system/konnectivity-agent-ff7kw"
Oct 31 13:02:44.711295 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.711295 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711235    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-device-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.711295 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711242    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-registration-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.711295 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711255    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-cni-binary-copy\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.711295 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711264    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-system-cni-dir\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.711295 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711291    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdl5n\" (UniqueName: \"kubernetes.io/projected/82689b74-1468-4087-8123-b361839b4124-kube-api-access-tdl5n\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.711863 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711319    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f97d160-4ad4-4546-9310-4a20d1309648-serviceca\") pod \"node-ca-nmwhw\" (UID: \"2f97d160-4ad4-4546-9310-4a20d1309648\") " pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:02:44.711863 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711452    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.711863 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711474    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.711863 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711612    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.711863 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711650    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-system-cni-dir\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.711863 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711702    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-device-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.711863 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.711772    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f97d160-4ad4-4546-9310-4a20d1309648-serviceca\") pod \"node-ca-nmwhw\" (UID: \"2f97d160-4ad4-4546-9310-4a20d1309648\") " pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:02:44.712279 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.712259    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.712279 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.712272    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1b19e39-5b5d-46f7-b005-a1e5165fe135-konnectivity-ca\") pod \"konnectivity-agent-ff7kw\" (UID: \"c1b19e39-5b5d-46f7-b005-a1e5165fe135\") " pod="kube-system/konnectivity-agent-ff7kw"
Oct 31 13:02:44.714695 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.714673    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1b19e39-5b5d-46f7-b005-a1e5165fe135-agent-certs\") pod \"konnectivity-agent-ff7kw\" (UID: \"c1b19e39-5b5d-46f7-b005-a1e5165fe135\") " pod="kube-system/konnectivity-agent-ff7kw"
Oct 31 13:02:44.724475 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:44.724452    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 13:02:44.724575 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:44.724477    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 13:02:44.724575 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:44.724500    2575 projected.go:196] Error preparing data for projected volume kube-api-access-kb2z7 for pod openshift-network-diagnostics/network-check-target-xpskx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:44.724575 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:44.724566    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7 podName:12e357d7-757d-4f89-837e-6283d4f87e11 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:45.22454832 +0000 UTC m=+3.135023547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kb2z7" (UniqueName: "kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7") pod "network-check-target-xpskx" (UID: "12e357d7-757d-4f89-837e-6283d4f87e11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:44.728946 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.728910    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmbrx\" (UniqueName: \"kubernetes.io/projected/8d617e36-a72a-4a8c-bc9c-85500f109e42-kube-api-access-bmbrx\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.729047 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.728917    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82sb9\" (UniqueName: \"kubernetes.io/projected/2f97d160-4ad4-4546-9310-4a20d1309648-kube-api-access-82sb9\") pod \"node-ca-nmwhw\" (UID: \"2f97d160-4ad4-4546-9310-4a20d1309648\") " pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:02:44.729115 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.729090    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhrzc\" (UniqueName: \"kubernetes.io/projected/0edfba66-6a3a-4be8-b420-227d0443ddea-kube-api-access-zhrzc\") pod \"network-metrics-daemon-s49pl\" (UID: \"0edfba66-6a3a-4be8-b420-227d0443ddea\") " pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:02:44.729332 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.729302    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdl5n\" (UniqueName: \"kubernetes.io/projected/82689b74-1468-4087-8123-b361839b4124-kube-api-access-tdl5n\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.818531 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.818496    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:02:44.820736 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.820708    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dxmqp"
Oct 31 13:02:44.826488 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.825122    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:02:44.831150 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.831123    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:02:44.833838 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:44.833782    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0b8381_9ec1_4e03_a025_d3f1d7ff7c46.slice/crio-4e10b9f7272d700c2e2165ef5d5d67e02e328a4f8ee9d5bfeb9817e15fe7e580 WatchSource:0}: Error finding container 4e10b9f7272d700c2e2165ef5d5d67e02e328a4f8ee9d5bfeb9817e15fe7e580: Status 404 returned error can't find the container with id 4e10b9f7272d700c2e2165ef5d5d67e02e328a4f8ee9d5bfeb9817e15fe7e580
Oct 31 13:02:44.838250 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:44.838223    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103c4c86_a9f6_4a59_a615_940717e39dc9.slice/crio-db3a7590fe5683b9717d7a2608a170530c7b18c2e42a2fabc409c9e290c0e357 WatchSource:0}: Error finding container db3a7590fe5683b9717d7a2608a170530c7b18c2e42a2fabc409c9e290c0e357: Status 404 returned error can't find the container with id db3a7590fe5683b9717d7a2608a170530c7b18c2e42a2fabc409c9e290c0e357
Oct 31 13:02:44.838820 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.838755    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:02:44.840059 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:44.840031    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24ca8d4_b321_4579_aff7_e1d0177f50bc.slice/crio-9a553509e10e565e44f362d84264a307b7baea6af3150ae2386065e0ebba4882 WatchSource:0}: Error finding container 9a553509e10e565e44f362d84264a307b7baea6af3150ae2386065e0ebba4882: Status 404 returned error can't find the container with id 9a553509e10e565e44f362d84264a307b7baea6af3150ae2386065e0ebba4882
Oct 31 13:02:44.843690 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:44.843666    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f1fbe3_7baa_4d79_b758_5c5cdc99f45d.slice/crio-4f3028dabf28d8ee66781a6f70ab02c40f38e5d1ab4040ab5c6c7e65b8925059 WatchSource:0}: Error finding container 4f3028dabf28d8ee66781a6f70ab02c40f38e5d1ab4040ab5c6c7e65b8925059: Status 404 returned error can't find the container with id 4f3028dabf28d8ee66781a6f70ab02c40f38e5d1ab4040ab5c6c7e65b8925059
Oct 31 13:02:44.848231 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.848211    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ff7kw"
Oct 31 13:02:44.850061 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:44.850026    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62aae01d_fd1d_43c3_9ad7_965bc6d595e2.slice/crio-edb3fe01cecc1e7d776497d6eb9af7760bb04ded5fe7de53cf0d97e7d547fda0 WatchSource:0}: Error finding container edb3fe01cecc1e7d776497d6eb9af7760bb04ded5fe7de53cf0d97e7d547fda0: Status 404 returned error can't find the container with id edb3fe01cecc1e7d776497d6eb9af7760bb04ded5fe7de53cf0d97e7d547fda0
Oct 31 13:02:44.858502 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:44.858468    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b19e39_5b5d_46f7_b005_a1e5165fe135.slice/crio-213c756f5bca005218873d47904eb58eefc7370d0138aa71a05d3fe97027d031 WatchSource:0}: Error finding container 213c756f5bca005218873d47904eb58eefc7370d0138aa71a05d3fe97027d031: Status 404 returned error can't find the container with id 213c756f5bca005218873d47904eb58eefc7370d0138aa71a05d3fe97027d031
Oct 31 13:02:44.863722 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.863704    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:02:44.875065 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:44.875036    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f97d160_4ad4_4546_9310_4a20d1309648.slice/crio-5f66ffa1bb68611960cb1b62e73cde35ee058af9bf249afb39a19dee9fedba87 WatchSource:0}: Error finding container 5f66ffa1bb68611960cb1b62e73cde35ee058af9bf249afb39a19dee9fedba87: Status 404 returned error can't find the container with id 5f66ffa1bb68611960cb1b62e73cde35ee058af9bf249afb39a19dee9fedba87
Oct 31 13:02:44.886807 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.886781    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:02:44.892187 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.892164    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:02:44.897651 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:44.897627    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d617e36_a72a_4a8c_bc9c_85500f109e42.slice/crio-30d4e2189000346936788ad998ca7abd115d9f270de12e6ec7ffa0e33f0ae607 WatchSource:0}: Error finding container 30d4e2189000346936788ad998ca7abd115d9f270de12e6ec7ffa0e33f0ae607: Status 404 returned error can't find the container with id 30d4e2189000346936788ad998ca7abd115d9f270de12e6ec7ffa0e33f0ae607
Oct 31 13:02:44.900176 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:44.900156    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:02:44.906574 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:02:44.906550    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82689b74_1468_4087_8123_b361839b4124.slice/crio-5c94c9269862aa3d9eee1072f8aa8c5c494e7de5d325a8b254e31eafc07eefc0 WatchSource:0}: Error finding container 5c94c9269862aa3d9eee1072f8aa8c5c494e7de5d325a8b254e31eafc07eefc0: Status 404 returned error can't find the container with id 5c94c9269862aa3d9eee1072f8aa8c5c494e7de5d325a8b254e31eafc07eefc0
Oct 31 13:02:45.114112 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:45.114008    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:45.114275 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:45.114197    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:45.114347 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:45.114285    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret podName:d7a89a36-092a-4d38-8a50-60619cc50832 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:46.114262009 +0000 UTC m=+4.024737211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret") pod "global-pull-secret-syncer-6tcxq" (UID: "d7a89a36-092a-4d38-8a50-60619cc50832") : object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:45.214800 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:45.214768    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs\") pod \"network-metrics-daemon-s49pl\" (UID: \"0edfba66-6a3a-4be8-b420-227d0443ddea\") " pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:02:45.214961 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:45.214896    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:45.215017 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:45.214969    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs podName:0edfba66-6a3a-4be8-b420-227d0443ddea nodeName:}" failed. No retries permitted until 2026-10-31 13:02:46.214954243 +0000 UTC m=+4.125429440 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs") pod "network-metrics-daemon-s49pl" (UID: "0edfba66-6a3a-4be8-b420-227d0443ddea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:45.315553 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:45.315520    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb2z7\" (UniqueName: \"kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7\") pod \"network-check-target-xpskx\" (UID: \"12e357d7-757d-4f89-837e-6283d4f87e11\") " pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:02:45.315712 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:45.315661    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 13:02:45.315712 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:45.315675    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 13:02:45.315712 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:45.315685    2575 projected.go:196] Error preparing data for projected volume kube-api-access-kb2z7 for pod openshift-network-diagnostics/network-check-target-xpskx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:45.315851 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:45.315747    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7 podName:12e357d7-757d-4f89-837e-6283d4f87e11 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:46.315728481 +0000 UTC m=+4.226203692 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kb2z7" (UniqueName: "kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7") pod "network-check-target-xpskx" (UID: "12e357d7-757d-4f89-837e-6283d4f87e11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:45.562980 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:45.554830    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:57:44 +0000 UTC" deadline="2028-04-29 00:04:09.414878074 +0000 UTC"
Oct 31 13:02:45.562980 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:45.554902    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13091h1m23.859979593s"
Oct 31 13:02:45.570592 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:45.570565    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:02:45.570733 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:45.570699    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpskx" podUID="12e357d7-757d-4f89-837e-6283d4f87e11"
Oct 31 13:02:45.587402 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:45.587268    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9" event={"ID":"103c4c86-a9f6-4a59-a615-940717e39dc9","Type":"ContainerStarted","Data":"db3a7590fe5683b9717d7a2608a170530c7b18c2e42a2fabc409c9e290c0e357"}
Oct 31 13:02:45.592484 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:45.592376    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6wvd" event={"ID":"8d617e36-a72a-4a8c-bc9c-85500f109e42","Type":"ContainerStarted","Data":"30d4e2189000346936788ad998ca7abd115d9f270de12e6ec7ffa0e33f0ae607"}
Oct 31 13:02:45.596266 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:45.596190    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7bpj5" event={"ID":"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d","Type":"ContainerStarted","Data":"4f3028dabf28d8ee66781a6f70ab02c40f38e5d1ab4040ab5c6c7e65b8925059"}
Oct 31 13:02:45.612002 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:45.611939    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dxmqp" event={"ID":"f24ca8d4-b321-4579-aff7-e1d0177f50bc","Type":"ContainerStarted","Data":"9a553509e10e565e44f362d84264a307b7baea6af3150ae2386065e0ebba4882"}
Oct 31 13:02:45.617670 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:45.617641    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rzs7m" event={"ID":"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46","Type":"ContainerStarted","Data":"4e10b9f7272d700c2e2165ef5d5d67e02e328a4f8ee9d5bfeb9817e15fe7e580"}
Oct 31 13:02:45.627801 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:45.627739    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9" event={"ID":"82689b74-1468-4087-8123-b361839b4124","Type":"ContainerStarted","Data":"5c94c9269862aa3d9eee1072f8aa8c5c494e7de5d325a8b254e31eafc07eefc0"}
Oct 31 13:02:45.641232 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:45.641165    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nmwhw" event={"ID":"2f97d160-4ad4-4546-9310-4a20d1309648","Type":"ContainerStarted","Data":"5f66ffa1bb68611960cb1b62e73cde35ee058af9bf249afb39a19dee9fedba87"}
Oct 31 13:02:45.644666 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:45.644600    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ff7kw" event={"ID":"c1b19e39-5b5d-46f7-b005-a1e5165fe135","Type":"ContainerStarted","Data":"213c756f5bca005218873d47904eb58eefc7370d0138aa71a05d3fe97027d031"}
Oct 31 13:02:45.648394 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:45.648266    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z65rw" event={"ID":"62aae01d-fd1d-43c3-9ad7-965bc6d595e2","Type":"ContainerStarted","Data":"edb3fe01cecc1e7d776497d6eb9af7760bb04ded5fe7de53cf0d97e7d547fda0"}
Oct 31 13:02:46.122825 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:46.122763    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:46.123036 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:46.123022    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:46.123166 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:46.123096    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret podName:d7a89a36-092a-4d38-8a50-60619cc50832 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:48.12307679 +0000 UTC m=+6.033552001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret") pod "global-pull-secret-syncer-6tcxq" (UID: "d7a89a36-092a-4d38-8a50-60619cc50832") : object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:46.223436 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:46.223399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs\") pod \"network-metrics-daemon-s49pl\" (UID: \"0edfba66-6a3a-4be8-b420-227d0443ddea\") " pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:02:46.223608 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:46.223562    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:46.223691 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:46.223625    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs podName:0edfba66-6a3a-4be8-b420-227d0443ddea nodeName:}" failed. No retries permitted until 2026-10-31 13:02:48.223606738 +0000 UTC m=+6.134081953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs") pod "network-metrics-daemon-s49pl" (UID: "0edfba66-6a3a-4be8-b420-227d0443ddea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:46.324965 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:46.324383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb2z7\" (UniqueName: \"kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7\") pod \"network-check-target-xpskx\" (UID: \"12e357d7-757d-4f89-837e-6283d4f87e11\") " pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:02:46.324965 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:46.324568    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 13:02:46.324965 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:46.324590    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 13:02:46.324965 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:46.324600    2575 projected.go:196] Error preparing data for projected volume kube-api-access-kb2z7 for pod openshift-network-diagnostics/network-check-target-xpskx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:46.324965 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:46.324644    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7 podName:12e357d7-757d-4f89-837e-6283d4f87e11 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:48.32463139 +0000 UTC m=+6.235106591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kb2z7" (UniqueName: "kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7") pod "network-check-target-xpskx" (UID: "12e357d7-757d-4f89-837e-6283d4f87e11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:46.555910 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:46.555852    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:57:44 +0000 UTC" deadline="2028-06-10 03:54:31.114166357 +0000 UTC"
Oct 31 13:02:46.555910 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:46.555909    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14102h51m44.558262475s"
Oct 31 13:02:46.570551 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:46.570040    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:02:46.570551 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:46.570184    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s49pl" podUID="0edfba66-6a3a-4be8-b420-227d0443ddea"
Oct 31 13:02:46.571441 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:46.571286    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:46.571441 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:46.571400    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tcxq" podUID="d7a89a36-092a-4d38-8a50-60619cc50832"
Oct 31 13:02:46.663889 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:46.663851    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal" event={"ID":"62681df534845faea6eef1fcf975d228","Type":"ContainerStarted","Data":"1197badf701b04f662e2cc986cf0daa70518d5d993136e0dddbb7e88947b40d3"}
Oct 31 13:02:46.681601 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:46.681532    2575 generic.go:356] "Generic (PLEG): container finished" podID="91e8205bd2503f1072bc57fd6d59d36b" containerID="d7b5039557c7f7f324f10ea016af84435d15f1b00d3de57d9b575d616fb26b65" exitCode=0
Oct 31 13:02:46.681761 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:46.681651    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal" event={"ID":"91e8205bd2503f1072bc57fd6d59d36b","Type":"ContainerDied","Data":"d7b5039557c7f7f324f10ea016af84435d15f1b00d3de57d9b575d616fb26b65"}
Oct 31 13:02:46.704414 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:46.703679    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal" podStartSLOduration=2.7036648359999997 podStartE2EDuration="2.703664836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:46.68384492 +0000 UTC m=+4.594320170" watchObservedRunningTime="2026-10-31 13:02:46.703664836 +0000 UTC m=+4.614140057"
Oct 31 13:02:47.570104 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:47.570056    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:02:47.570104 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:47.570197    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpskx" podUID="12e357d7-757d-4f89-837e-6283d4f87e11"
Oct 31 13:02:47.701178 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:47.701013    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal" event={"ID":"91e8205bd2503f1072bc57fd6d59d36b","Type":"ContainerStarted","Data":"6308a07606b9fc75bdc1f8a01adaff0c39fe66ac3f32d4c6bbd073b682d1dd56"}
Oct 31 13:02:47.732895 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:47.731216    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal" podStartSLOduration=3.731195649 podStartE2EDuration="3.731195649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:47.731172745 +0000 UTC m=+5.641647995" watchObservedRunningTime="2026-10-31 13:02:47.731195649 +0000 UTC m=+5.641670871"
Oct 31 13:02:48.138598 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:48.138561    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:48.138736 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:48.138697    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:48.138809 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:48.138764    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret podName:d7a89a36-092a-4d38-8a50-60619cc50832 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:52.138743899 +0000 UTC m=+10.049219112 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret") pod "global-pull-secret-syncer-6tcxq" (UID: "d7a89a36-092a-4d38-8a50-60619cc50832") : object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:48.238982 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:48.238939    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs\") pod \"network-metrics-daemon-s49pl\" (UID: \"0edfba66-6a3a-4be8-b420-227d0443ddea\") " pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:02:48.239180 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:48.239108    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:48.239180 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:48.239181    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs podName:0edfba66-6a3a-4be8-b420-227d0443ddea nodeName:}" failed. No retries permitted until 2026-10-31 13:02:52.239158128 +0000 UTC m=+10.149633333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs") pod "network-metrics-daemon-s49pl" (UID: "0edfba66-6a3a-4be8-b420-227d0443ddea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:48.339794 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:48.339651    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb2z7\" (UniqueName: \"kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7\") pod \"network-check-target-xpskx\" (UID: \"12e357d7-757d-4f89-837e-6283d4f87e11\") " pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:02:48.339980 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:48.339877    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 13:02:48.339980 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:48.339922    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 13:02:48.339980 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:48.339937    2575 projected.go:196] Error preparing data for projected volume kube-api-access-kb2z7 for pod openshift-network-diagnostics/network-check-target-xpskx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:48.340164 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:48.340021    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7 podName:12e357d7-757d-4f89-837e-6283d4f87e11 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:52.340001442 +0000 UTC m=+10.250476656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kb2z7" (UniqueName: "kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7") pod "network-check-target-xpskx" (UID: "12e357d7-757d-4f89-837e-6283d4f87e11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:48.570454 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:48.570366    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:48.570611 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:48.570512    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tcxq" podUID="d7a89a36-092a-4d38-8a50-60619cc50832"
Oct 31 13:02:48.570986 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:48.570956    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:02:48.571111 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:48.571071    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s49pl" podUID="0edfba66-6a3a-4be8-b420-227d0443ddea"
Oct 31 13:02:48.715340 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:48.715298    2575 generic.go:356] "Generic (PLEG): container finished" podID="8d617e36-a72a-4a8c-bc9c-85500f109e42" containerID="36caeef49cd719e6dd70cd18ce0b0cee2f25808d95a629bdaf553a2f57b0a5d7" exitCode=0
Oct 31 13:02:48.716610 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:48.716584    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6wvd" event={"ID":"8d617e36-a72a-4a8c-bc9c-85500f109e42","Type":"ContainerDied","Data":"36caeef49cd719e6dd70cd18ce0b0cee2f25808d95a629bdaf553a2f57b0a5d7"}
Oct 31 13:02:49.570070 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:49.570036    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:02:49.570260 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:49.570165    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpskx" podUID="12e357d7-757d-4f89-837e-6283d4f87e11"
Oct 31 13:02:49.732852 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:49.732014    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ff7kw" event={"ID":"c1b19e39-5b5d-46f7-b005-a1e5165fe135","Type":"ContainerStarted","Data":"8e68565a4ba32e4c515974a0dc151435455b89abc0a28bead72510d92ff622ec"}
Oct 31 13:02:50.265625 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:50.265556    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ff7kw"
Oct 31 13:02:50.266111 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:50.266091    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ff7kw"
Oct 31 13:02:50.285278 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:50.285215    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ff7kw" podStartSLOduration=3.034349583 podStartE2EDuration="7.285197454s" totalImagesPullingTime="4.250847871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:49.759072584 +0000 UTC m=+7.669547804" watchObservedRunningTime="2026-10-31 13:02:50.285197454 +0000 UTC m=+8.195672686"
Oct 31 13:02:50.572094 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:50.572016    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:02:50.572254 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:50.572162    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s49pl" podUID="0edfba66-6a3a-4be8-b420-227d0443ddea"
Oct 31 13:02:50.572541 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:50.572521    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:50.572637 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:50.572616    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tcxq" podUID="d7a89a36-092a-4d38-8a50-60619cc50832"
Oct 31 13:02:50.738108 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:50.737609    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nmwhw" event={"ID":"2f97d160-4ad4-4546-9310-4a20d1309648","Type":"ContainerStarted","Data":"1bcdce426886df1585ac19962abff4ed63b83807ad6bfd0d3508bea6da01f8ff"}
Oct 31 13:02:50.738108 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:50.737708    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ff7kw"
Oct 31 13:02:50.738913 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:50.738888    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ff7kw"
Oct 31 13:02:50.783411 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:50.783351    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nmwhw" podStartSLOduration=2.677063694 podStartE2EDuration="7.783333877s" totalImagesPullingTime="5.106270183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:50.757375114 +0000 UTC m=+8.667850349" watchObservedRunningTime="2026-10-31 13:02:50.783333877 +0000 UTC m=+8.693809098"
Oct 31 13:02:51.570686 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:51.570643    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:02:51.570928 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:51.570788    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpskx" podUID="12e357d7-757d-4f89-837e-6283d4f87e11"
Oct 31 13:02:51.742845 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:51.742807    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9" event={"ID":"82689b74-1468-4087-8123-b361839b4124","Type":"ContainerStarted","Data":"604366bc264c63f3b69e363729ec3459cf7b9f012179f02d9ea732cbb9974fe1"}
Oct 31 13:02:52.175754 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:52.175714    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:52.175951 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:52.175900    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:52.176048 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:52.175973    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret podName:d7a89a36-092a-4d38-8a50-60619cc50832 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:00.175956091 +0000 UTC m=+18.086431301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret") pod "global-pull-secret-syncer-6tcxq" (UID: "d7a89a36-092a-4d38-8a50-60619cc50832") : object "kube-system"/"original-pull-secret" not registered
Oct 31 13:02:52.276644 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:52.276594    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs\") pod \"network-metrics-daemon-s49pl\" (UID: \"0edfba66-6a3a-4be8-b420-227d0443ddea\") " pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:02:52.276826 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:52.276739    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:52.276826 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:52.276801    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs podName:0edfba66-6a3a-4be8-b420-227d0443ddea nodeName:}" failed. No retries permitted until 2026-10-31 13:03:00.276784023 +0000 UTC m=+18.187259226 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs") pod "network-metrics-daemon-s49pl" (UID: "0edfba66-6a3a-4be8-b420-227d0443ddea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:02:52.377285 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:52.377245    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb2z7\" (UniqueName: \"kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7\") pod \"network-check-target-xpskx\" (UID: \"12e357d7-757d-4f89-837e-6283d4f87e11\") " pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:02:52.377472 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:52.377456    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 13:02:52.377550 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:52.377478    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 13:02:52.377550 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:52.377492    2575 projected.go:196] Error preparing data for projected volume kube-api-access-kb2z7 for pod openshift-network-diagnostics/network-check-target-xpskx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:52.377550 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:52.377548    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7 podName:12e357d7-757d-4f89-837e-6283d4f87e11 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:00.377531574 +0000 UTC m=+18.288006792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kb2z7" (UniqueName: "kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7") pod "network-check-target-xpskx" (UID: "12e357d7-757d-4f89-837e-6283d4f87e11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:02:52.572909 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:52.572848    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:52.573065 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:52.572992    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tcxq" podUID="d7a89a36-092a-4d38-8a50-60619cc50832"
Oct 31 13:02:52.573263 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:52.573241    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:02:52.573378 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:52.573356    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s49pl" podUID="0edfba66-6a3a-4be8-b420-227d0443ddea"
Oct 31 13:02:52.647678 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:52.647612    2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 13:02:52.749484 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:52.748820    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dxmqp" event={"ID":"f24ca8d4-b321-4579-aff7-e1d0177f50bc","Type":"ContainerStarted","Data":"ae20f5ee9b62eef02b2af84b1f1cc7793388e50af764bfec96c6b5f50f02552b"}
Oct 31 13:02:52.757937 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:52.755967    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9" event={"ID":"82689b74-1468-4087-8123-b361839b4124","Type":"ContainerStarted","Data":"368bcc72a30b3320f14b2c9ddd28af1b71b65d61564fb6d81d0b8e15346716a9"}
Oct 31 13:02:53.568646 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:53.568335    2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T13:02:52.647627923Z","UUID":"17170a6b-f146-4c57-a435-10baff16b90a","Handler":null,"Name":"","Endpoint":""}
Oct 31 13:02:53.569967 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:53.569834    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 13:02:53.569967 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:53.569897    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 13:02:53.570183 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:53.569858    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:02:53.570183 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:53.570148    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpskx" podUID="12e357d7-757d-4f89-837e-6283d4f87e11"
Oct 31 13:02:53.760048 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:53.760012    2575 generic.go:356] "Generic (PLEG): container finished" podID="8d617e36-a72a-4a8c-bc9c-85500f109e42" containerID="b3d75dd886e3341305bd56487c668509b50d9a29015dffc2d982ca147197fb87" exitCode=0
Oct 31 13:02:53.760446 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:53.760112    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6wvd" event={"ID":"8d617e36-a72a-4a8c-bc9c-85500f109e42","Type":"ContainerDied","Data":"b3d75dd886e3341305bd56487c668509b50d9a29015dffc2d982ca147197fb87"}
Oct 31 13:02:53.762943 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:53.762915    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9" event={"ID":"82689b74-1468-4087-8123-b361839b4124","Type":"ContainerStarted","Data":"902e8d5d67b5bbbd68bc5245c2bfaf947b682103730d122a0a3033b2329718f3"}
Oct 31 13:02:53.764626 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:53.764598    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z65rw" event={"ID":"62aae01d-fd1d-43c3-9ad7-965bc6d595e2","Type":"ContainerStarted","Data":"34f893c33a54246c6adcd6f67176cd9161e419d5e6e740d00b61cf2757e35b15"}
Oct 31 13:02:53.785657 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:53.785580    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-dxmqp" podStartSLOduration=2.986029264 podStartE2EDuration="10.785566321s" totalImagesPullingTime="7.799537057s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:52.778538714 +0000 UTC m=+10.689013942" watchObservedRunningTime="2026-10-31 13:02:53.785566321 +0000 UTC m=+11.696041539"
Oct 31 13:02:53.809014 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:53.808976    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9" podStartSLOduration=2.437848374 podStartE2EDuration="10.808964043s" totalImagesPullingTime="8.371115669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:53.808697239 +0000 UTC m=+11.719172492" watchObservedRunningTime="2026-10-31 13:02:53.808964043 +0000 UTC m=+11.719439260"
Oct 31 13:02:53.827445 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:53.827388    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-z65rw" podStartSLOduration=2.566436803 podStartE2EDuration="10.827373312s" totalImagesPullingTime="8.260936509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:53.827026295 +0000 UTC m=+11.737501516" watchObservedRunningTime="2026-10-31 13:02:53.827373312 +0000 UTC m=+11.737848530"
Oct 31 13:02:54.570179 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:54.570146    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:54.570308 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:54.570286    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tcxq" podUID="d7a89a36-092a-4d38-8a50-60619cc50832"
Oct 31 13:02:54.570362 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:54.570343    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:02:54.570487 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:54.570463    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s49pl" podUID="0edfba66-6a3a-4be8-b420-227d0443ddea"
Oct 31 13:02:54.769197 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:54.769155    2575 generic.go:356] "Generic (PLEG): container finished" podID="8d617e36-a72a-4a8c-bc9c-85500f109e42" containerID="f208a9e7998f2a5cb093ca67c4d755403298508c1d986c7aa1ad600f03b4e0ee" exitCode=0
Oct 31 13:02:54.769722 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:54.769212    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6wvd" event={"ID":"8d617e36-a72a-4a8c-bc9c-85500f109e42","Type":"ContainerDied","Data":"f208a9e7998f2a5cb093ca67c4d755403298508c1d986c7aa1ad600f03b4e0ee"}
Oct 31 13:02:54.771837 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:54.771811    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7bpj5" event={"ID":"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d","Type":"ContainerStarted","Data":"5c701e3ebebe7f333a6d5d620d9adc633908d71ab3b48fa609dfaccba78f489c"}
Oct 31 13:02:54.816323 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:54.816284    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7bpj5" podStartSLOduration=3.691383086 podStartE2EDuration="11.816272363s" totalImagesPullingTime="8.124889277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:54.816153922 +0000 UTC m=+12.726629143" watchObservedRunningTime="2026-10-31 13:02:54.816272363 +0000 UTC m=+12.726747591"
Oct 31 13:02:55.569683 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:55.569661    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:02:55.569796 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:55.569774    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpskx" podUID="12e357d7-757d-4f89-837e-6283d4f87e11"
Oct 31 13:02:55.776487 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:55.776414    2575 generic.go:356] "Generic (PLEG): container finished" podID="8d617e36-a72a-4a8c-bc9c-85500f109e42" containerID="f505b7ed1b44378b61fc85db85953c9cdab8e8d15ba1b9efb119cccdf08de270" exitCode=0
Oct 31 13:02:55.777126 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:55.776505    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6wvd" event={"ID":"8d617e36-a72a-4a8c-bc9c-85500f109e42","Type":"ContainerDied","Data":"f505b7ed1b44378b61fc85db85953c9cdab8e8d15ba1b9efb119cccdf08de270"}
Oct 31 13:02:55.778179 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:55.778149    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rzs7m" event={"ID":"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46","Type":"ContainerStarted","Data":"df347c71cf5c9daeee549404e39022163b8678515331d376bd7cbb527ee68387"}
Oct 31 13:02:56.570208 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:56.570175    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:02:56.570418 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:56.570377    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s49pl" podUID="0edfba66-6a3a-4be8-b420-227d0443ddea"
Oct 31 13:02:56.570418 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:56.570394    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:56.570585 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:56.570502    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tcxq" podUID="d7a89a36-092a-4d38-8a50-60619cc50832"
Oct 31 13:02:57.570451 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:57.570419    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:02:57.571061 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:57.570542    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpskx" podUID="12e357d7-757d-4f89-837e-6283d4f87e11"
Oct 31 13:02:58.570126 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:58.569918    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:02:58.570359 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:58.569968    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:02:58.570359 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:58.570226    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tcxq" podUID="d7a89a36-092a-4d38-8a50-60619cc50832"
Oct 31 13:02:58.570359 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:58.570307    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s49pl" podUID="0edfba66-6a3a-4be8-b420-227d0443ddea"
Oct 31 13:02:58.787538 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:58.787501    2575 generic.go:356] "Generic (PLEG): container finished" podID="8d617e36-a72a-4a8c-bc9c-85500f109e42" containerID="8239bb306218a8c34e9a8211de5baf8047690d25913007a9f3c1914487ac4407" exitCode=0
Oct 31 13:02:58.787953 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:58.787576    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6wvd" event={"ID":"8d617e36-a72a-4a8c-bc9c-85500f109e42","Type":"ContainerDied","Data":"8239bb306218a8c34e9a8211de5baf8047690d25913007a9f3c1914487ac4407"}
Oct 31 13:02:58.816587 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:58.816531    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rzs7m" podStartSLOduration=5.514193539 podStartE2EDuration="15.816510491s" totalImagesPullingTime="10.302316952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:55.827461988 +0000 UTC m=+13.737937205" watchObservedRunningTime="2026-10-31 13:02:58.816510491 +0000 UTC m=+16.726985710"
Oct 31 13:02:59.569838 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:59.569800    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:02:59.570029 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:02:59.569933    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpskx" podUID="12e357d7-757d-4f89-837e-6283d4f87e11"
Oct 31 13:02:59.794782 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:59.794122    2575 generic.go:356] "Generic (PLEG): container finished" podID="8d617e36-a72a-4a8c-bc9c-85500f109e42" containerID="80b51cbabe9267284425d3b962dd83d284b59cc2e1800546c0be1c273232ac57" exitCode=0
Oct 31 13:02:59.794782 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:02:59.794189    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6wvd" event={"ID":"8d617e36-a72a-4a8c-bc9c-85500f109e42","Type":"ContainerDied","Data":"80b51cbabe9267284425d3b962dd83d284b59cc2e1800546c0be1c273232ac57"}
Oct 31 13:03:00.236336 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:00.236304    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:00.236497 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:00.236443    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 13:03:00.236555 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:00.236510    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret podName:d7a89a36-092a-4d38-8a50-60619cc50832 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:16.236496559 +0000 UTC m=+34.146971765 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret") pod "global-pull-secret-syncer-6tcxq" (UID: "d7a89a36-092a-4d38-8a50-60619cc50832") : object "kube-system"/"original-pull-secret" not registered
Oct 31 13:03:00.337312 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:00.337258    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs\") pod \"network-metrics-daemon-s49pl\" (UID: \"0edfba66-6a3a-4be8-b420-227d0443ddea\") " pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:03:00.337485 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:00.337422    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:03:00.337485 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:00.337484    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs podName:0edfba66-6a3a-4be8-b420-227d0443ddea nodeName:}" failed. No retries permitted until 2026-10-31 13:03:16.337467159 +0000 UTC m=+34.247942357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs") pod "network-metrics-daemon-s49pl" (UID: "0edfba66-6a3a-4be8-b420-227d0443ddea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 13:03:00.437811 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:00.437770    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb2z7\" (UniqueName: \"kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7\") pod \"network-check-target-xpskx\" (UID: \"12e357d7-757d-4f89-837e-6283d4f87e11\") " pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:03:00.438054 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:00.438036    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 13:03:00.438111 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:00.438062    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 13:03:00.438111 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:00.438082    2575 projected.go:196] Error preparing data for projected volume kube-api-access-kb2z7 for pod openshift-network-diagnostics/network-check-target-xpskx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:03:00.438208 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:00.438143    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7 podName:12e357d7-757d-4f89-837e-6283d4f87e11 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:16.43811899 +0000 UTC m=+34.348594192 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kb2z7" (UniqueName: "kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7") pod "network-check-target-xpskx" (UID: "12e357d7-757d-4f89-837e-6283d4f87e11") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 13:03:00.570025 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:00.569949    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:00.570025 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:00.569992    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:03:00.570218 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:00.570087    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tcxq" podUID="d7a89a36-092a-4d38-8a50-60619cc50832"
Oct 31 13:03:00.570281 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:00.570238    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s49pl" podUID="0edfba66-6a3a-4be8-b420-227d0443ddea"
Oct 31 13:03:00.800677 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:00.800636    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x6wvd" event={"ID":"8d617e36-a72a-4a8c-bc9c-85500f109e42","Type":"ContainerStarted","Data":"272c1707a5f212b39c400f18265fc1c61a3a08287ac8c8dec6042ec407d312e6"}
Oct 31 13:03:00.831571 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:00.831483    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x6wvd" podStartSLOduration=7.099739824 podStartE2EDuration="17.83146589s" totalImagesPullingTime="10.731726066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:00.829825473 +0000 UTC m=+18.740300690" watchObservedRunningTime="2026-10-31 13:03:00.83146589 +0000 UTC m=+18.741941109"
Oct 31 13:03:01.570727 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:01.570551    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:03:01.570851 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:01.570829    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpskx" podUID="12e357d7-757d-4f89-837e-6283d4f87e11"
Oct 31 13:03:01.805723 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:01.805685    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9" event={"ID":"103c4c86-a9f6-4a59-a615-940717e39dc9","Type":"ContainerStarted","Data":"f06cc28d608cb03ab8054416f9b29ca0c8505b4baf346b6f33c81bdae246c9fb"}
Oct 31 13:03:01.806109 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:01.805768    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9" event={"ID":"103c4c86-a9f6-4a59-a615-940717e39dc9","Type":"ContainerStarted","Data":"b76acd208b5380e8d2d0da78db045f771bf8e48976b75fc85dc583a777d491d5"}
Oct 31 13:03:01.806109 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:01.805801    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9" event={"ID":"103c4c86-a9f6-4a59-a615-940717e39dc9","Type":"ContainerStarted","Data":"ad32f5c1e96d52d3b8e4d2965e15421015cb15f4217f45accaf17d39a959b6ae"}
Oct 31 13:03:01.806109 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:01.805840    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9" event={"ID":"103c4c86-a9f6-4a59-a615-940717e39dc9","Type":"ContainerStarted","Data":"ee228bb9bf4f80e33c7ba84f02ac4700bc3b33b960abcb7e1bdea783aaa6ae44"}
Oct 31 13:03:01.806109 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:01.805885    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9" event={"ID":"103c4c86-a9f6-4a59-a615-940717e39dc9","Type":"ContainerStarted","Data":"3a45dfd908181cbf9d0f01a8d4cfb37f67c00597d324bd7fdb8fb9221c9599f4"}
Oct 31 13:03:01.806109 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:01.805925    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9" event={"ID":"103c4c86-a9f6-4a59-a615-940717e39dc9","Type":"ContainerStarted","Data":"2271f1944a0dccb42f47afe314094e043b07374ee90ad8b9ad0730086d3ecc00"}
Oct 31 13:03:02.570710 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:02.570668    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:03:02.570884 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:02.570793    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s49pl" podUID="0edfba66-6a3a-4be8-b420-227d0443ddea"
Oct 31 13:03:02.570884 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:02.570852    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:02.570986 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:02.570974    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tcxq" podUID="d7a89a36-092a-4d38-8a50-60619cc50832"
Oct 31 13:03:03.570488 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:03.570455    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:03:03.570891 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:03.570552    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpskx" podUID="12e357d7-757d-4f89-837e-6283d4f87e11"
Oct 31 13:03:03.812301 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:03.812268    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9" event={"ID":"103c4c86-a9f6-4a59-a615-940717e39dc9","Type":"ContainerStarted","Data":"4e806060d3c39228fef1aae0892d0d73cbf169bdc9081ce273229e148c731b5a"}
Oct 31 13:03:04.569762 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:04.569726    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:04.570028 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:04.569838    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tcxq" podUID="d7a89a36-092a-4d38-8a50-60619cc50832"
Oct 31 13:03:04.570028 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:04.569895    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:03:04.570028 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:04.569992    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s49pl" podUID="0edfba66-6a3a-4be8-b420-227d0443ddea"
Oct 31 13:03:05.570063 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:05.570034    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:03:05.570612 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:05.570124    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpskx" podUID="12e357d7-757d-4f89-837e-6283d4f87e11"
Oct 31 13:03:06.570057 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:06.570027    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:06.570313 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:06.570033    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:03:06.570313 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:06.570156    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tcxq" podUID="d7a89a36-092a-4d38-8a50-60619cc50832"
Oct 31 13:03:06.570313 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:06.570208    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s49pl" podUID="0edfba66-6a3a-4be8-b420-227d0443ddea"
Oct 31 13:03:06.821247 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:06.821170    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9" event={"ID":"103c4c86-a9f6-4a59-a615-940717e39dc9","Type":"ContainerStarted","Data":"727d937d778b09e5075a56473d9178c320992d0d0c3dc5b81f5cfe9539a1ff7f"}
Oct 31 13:03:06.821483 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:06.821450    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:06.821603 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:06.821527    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:06.837600 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:06.837575    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:06.837724 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:06.837668    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:06.857649 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:06.857611    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9" podStartSLOduration=7.287885588 podStartE2EDuration="23.857600918s" totalImagesPullingTime="16.56971533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:06.857139152 +0000 UTC m=+24.767614370" watchObservedRunningTime="2026-10-31 13:03:06.857600918 +0000 UTC m=+24.768076135"
Oct 31 13:03:07.569541 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:07.569514    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:03:07.569697 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:07.569635    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpskx" podUID="12e357d7-757d-4f89-837e-6283d4f87e11"
Oct 31 13:03:07.610423 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:07.610397    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6tcxq"]
Oct 31 13:03:07.610846 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:07.610535    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:07.610846 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:07.610649    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tcxq" podUID="d7a89a36-092a-4d38-8a50-60619cc50832"
Oct 31 13:03:07.611330 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:07.611309    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s49pl"]
Oct 31 13:03:07.611442 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:07.611428    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:03:07.611562 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:07.611542    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s49pl" podUID="0edfba66-6a3a-4be8-b420-227d0443ddea"
Oct 31 13:03:07.612557 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:07.612532    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xpskx"]
Oct 31 13:03:07.823327 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:07.823298    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:03:07.823515 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:07.823370    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 13:03:07.823515 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:07.823414    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpskx" podUID="12e357d7-757d-4f89-837e-6283d4f87e11"
Oct 31 13:03:08.530126 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:08.529946    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:09.570138 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:09.570108    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:09.570613 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:09.570115    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:03:09.570613 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:09.570221    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tcxq" podUID="d7a89a36-092a-4d38-8a50-60619cc50832"
Oct 31 13:03:09.570613 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:09.570115    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:03:09.570613 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:09.570292    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s49pl" podUID="0edfba66-6a3a-4be8-b420-227d0443ddea"
Oct 31 13:03:09.570613 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:09.570344    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpskx" podUID="12e357d7-757d-4f89-837e-6283d4f87e11"
Oct 31 13:03:11.570316 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:11.570277    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:03:11.570743 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:11.570321    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:11.570743 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:11.570327    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:03:11.570743 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:11.570396    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpskx" podUID="12e357d7-757d-4f89-837e-6283d4f87e11"
Oct 31 13:03:11.570743 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:11.570487    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tcxq" podUID="d7a89a36-092a-4d38-8a50-60619cc50832"
Oct 31 13:03:11.570743 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:11.570569    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s49pl" podUID="0edfba66-6a3a-4be8-b420-227d0443ddea"
Oct 31 13:03:12.754285 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:12.754253    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:12.979115 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:12.979086    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:13.424056 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.423973    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeReady"
Oct 31 13:03:13.424212 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.424114    2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 13:03:13.441816 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.441785    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeReady"
Oct 31 13:03:13.473798 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.473770    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kstrj"]
Oct 31 13:03:13.514086 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.514059    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jnzs8"]
Oct 31 13:03:13.514243 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.514199    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:13.518601 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.518573    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 13:03:13.518755 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.518741    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 13:03:13.518934 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.518918    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qbkfg\""
Oct 31 13:03:13.529344 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.529325    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kstrj"]
Oct 31 13:03:13.529444 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.529364    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jnzs8"]
Oct 31 13:03:13.529444 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.529419    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jnzs8"
Oct 31 13:03:13.569855 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.569827    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:13.569983 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.569859    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:03:13.569983 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.569861    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:03:13.581122 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.581076    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:13.619978 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.619950    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 13:03:13.620095 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.620026    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 13:03:13.620095 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.620029    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 13:03:13.620186 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.620089    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 13:03:13.620256 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.620236    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 13:03:13.620359 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.620274    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 13:03:13.620359 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.620284    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wr5k7\""
Oct 31 13:03:13.620359 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.620333    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j8xvf\""
Oct 31 13:03:13.621405 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.621361    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5rqjw\""
Oct 31 13:03:13.621625 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.621605    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wdnw7\""
Oct 31 13:03:13.621700 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.621635    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 13:03:13.629876 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.629842    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb046f36-d918-4cdd-b2be-f2eec97ae74e-config-volume\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:13.629958 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.629888    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb046f36-d918-4cdd-b2be-f2eec97ae74e-tmp-dir\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:13.629958 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.629924    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rv7\" (UniqueName: \"kubernetes.io/projected/575eac51-2cde-4d41-9de7-f6987fb83dfe-kube-api-access-64rv7\") pod \"ingress-canary-jnzs8\" (UID: \"575eac51-2cde-4d41-9de7-f6987fb83dfe\") " pod="openshift-ingress-canary/ingress-canary-jnzs8"
Oct 31 13:03:13.630052 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.630037    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/575eac51-2cde-4d41-9de7-f6987fb83dfe-cert\") pod \"ingress-canary-jnzs8\" (UID: \"575eac51-2cde-4d41-9de7-f6987fb83dfe\") " pod="openshift-ingress-canary/ingress-canary-jnzs8"
Oct 31 13:03:13.630085 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.630067    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rnl5\" (UniqueName: \"kubernetes.io/projected/cb046f36-d918-4cdd-b2be-f2eec97ae74e-kube-api-access-9rnl5\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:13.630185 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.630169    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:13.731141 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.731109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb046f36-d918-4cdd-b2be-f2eec97ae74e-config-volume\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:13.731276 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.731147    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb046f36-d918-4cdd-b2be-f2eec97ae74e-tmp-dir\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:13.731276 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.731186    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rv7\" (UniqueName: \"kubernetes.io/projected/575eac51-2cde-4d41-9de7-f6987fb83dfe-kube-api-access-64rv7\") pod \"ingress-canary-jnzs8\" (UID: \"575eac51-2cde-4d41-9de7-f6987fb83dfe\") " pod="openshift-ingress-canary/ingress-canary-jnzs8"
Oct 31 13:03:13.731276 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.731210    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/575eac51-2cde-4d41-9de7-f6987fb83dfe-cert\") pod \"ingress-canary-jnzs8\" (UID: \"575eac51-2cde-4d41-9de7-f6987fb83dfe\") " pod="openshift-ingress-canary/ingress-canary-jnzs8"
Oct 31 13:03:13.731276 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.731234    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rnl5\" (UniqueName: \"kubernetes.io/projected/cb046f36-d918-4cdd-b2be-f2eec97ae74e-kube-api-access-9rnl5\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:13.731462 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.731399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:13.731566 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:13.731530    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 13:03:13.731566 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.731555    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb046f36-d918-4cdd-b2be-f2eec97ae74e-tmp-dir\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:13.731707 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:13.731589    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls podName:cb046f36-d918-4cdd-b2be-f2eec97ae74e nodeName:}" failed. No retries permitted until 2026-10-31 13:03:14.23157347 +0000 UTC m=+32.142048665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls") pod "dns-default-kstrj" (UID: "cb046f36-d918-4cdd-b2be-f2eec97ae74e") : secret "dns-default-metrics-tls" not found
Oct 31 13:03:13.731808 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.731785    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb046f36-d918-4cdd-b2be-f2eec97ae74e-config-volume\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:13.734602 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.734582    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/575eac51-2cde-4d41-9de7-f6987fb83dfe-cert\") pod \"ingress-canary-jnzs8\" (UID: \"575eac51-2cde-4d41-9de7-f6987fb83dfe\") " pod="openshift-ingress-canary/ingress-canary-jnzs8"
Oct 31 13:03:13.735471 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.735451    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:13.744606 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.744580    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rnl5\" (UniqueName: \"kubernetes.io/projected/cb046f36-d918-4cdd-b2be-f2eec97ae74e-kube-api-access-9rnl5\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:13.744708 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.744692    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rv7\" (UniqueName: \"kubernetes.io/projected/575eac51-2cde-4d41-9de7-f6987fb83dfe-kube-api-access-64rv7\") pod \"ingress-canary-jnzs8\" (UID: \"575eac51-2cde-4d41-9de7-f6987fb83dfe\") " pod="openshift-ingress-canary/ingress-canary-jnzs8"
Oct 31 13:03:13.854469 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:13.854440    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jnzs8"
Oct 31 13:03:14.196217 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:14.196138    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jnzs8"]
Oct 31 13:03:14.199225 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:03:14.199195    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575eac51_2cde_4d41_9de7_f6987fb83dfe.slice/crio-be87dc68656727a978cb575eb654c681df63fe90007912a1d65ee06435c10d95 WatchSource:0}: Error finding container be87dc68656727a978cb575eb654c681df63fe90007912a1d65ee06435c10d95: Status 404 returned error can't find the container with id be87dc68656727a978cb575eb654c681df63fe90007912a1d65ee06435c10d95
Oct 31 13:03:14.233924 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:14.233895    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:14.234051 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:14.234032    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 13:03:14.234094 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:14.234089    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls podName:cb046f36-d918-4cdd-b2be-f2eec97ae74e nodeName:}" failed. No retries permitted until 2026-10-31 13:03:15.234076047 +0000 UTC m=+33.144551279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls") pod "dns-default-kstrj" (UID: "cb046f36-d918-4cdd-b2be-f2eec97ae74e") : secret "dns-default-metrics-tls" not found
Oct 31 13:03:14.840967 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:14.840926    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jnzs8" event={"ID":"575eac51-2cde-4d41-9de7-f6987fb83dfe","Type":"ContainerStarted","Data":"be87dc68656727a978cb575eb654c681df63fe90007912a1d65ee06435c10d95"}
Oct 31 13:03:15.240863 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:15.240825    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:15.241292 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:15.240979    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 13:03:15.241292 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:15.241045    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls podName:cb046f36-d918-4cdd-b2be-f2eec97ae74e nodeName:}" failed. No retries permitted until 2026-10-31 13:03:17.241026722 +0000 UTC m=+35.151501937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls") pod "dns-default-kstrj" (UID: "cb046f36-d918-4cdd-b2be-f2eec97ae74e") : secret "dns-default-metrics-tls" not found
Oct 31 13:03:16.249910 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:16.249854    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:16.252793 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:16.252769    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:16.304904 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:16.304854    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:16.350577 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:16.350543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs\") pod \"network-metrics-daemon-s49pl\" (UID: \"0edfba66-6a3a-4be8-b420-227d0443ddea\") " pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:03:16.350736 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:16.350662    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 13:03:16.350736 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:16.350724    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs podName:0edfba66-6a3a-4be8-b420-227d0443ddea nodeName:}" failed. No retries permitted until 2026-10-31 13:03:48.350708043 +0000 UTC m=+66.261183256 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs") pod "network-metrics-daemon-s49pl" (UID: "0edfba66-6a3a-4be8-b420-227d0443ddea") : secret "metrics-daemon-secret" not found
Oct 31 13:03:16.451503 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:16.451464    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb2z7\" (UniqueName: \"kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7\") pod \"network-check-target-xpskx\" (UID: \"12e357d7-757d-4f89-837e-6283d4f87e11\") " pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:03:16.454206 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:16.454185    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb2z7\" (UniqueName: \"kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7\") pod \"network-check-target-xpskx\" (UID: \"12e357d7-757d-4f89-837e-6283d4f87e11\") " pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:03:16.608113 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:16.608041    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:03:16.631493 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:16.631469    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6tcxq"]
Oct 31 13:03:16.634744 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:03:16.634710    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a89a36_092a_4d38_8a50_60619cc50832.slice/crio-093df186420655fb58f0dd5c6e8b6da524689a7306f29de04d4e28fd8f401f0b WatchSource:0}: Error finding container 093df186420655fb58f0dd5c6e8b6da524689a7306f29de04d4e28fd8f401f0b: Status 404 returned error can't find the container with id 093df186420655fb58f0dd5c6e8b6da524689a7306f29de04d4e28fd8f401f0b
Oct 31 13:03:16.846708 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:16.846680    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6tcxq" event={"ID":"d7a89a36-092a-4d38-8a50-60619cc50832","Type":"ContainerStarted","Data":"093df186420655fb58f0dd5c6e8b6da524689a7306f29de04d4e28fd8f401f0b"}
Oct 31 13:03:16.899798 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:16.899778    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xpskx"]
Oct 31 13:03:16.902135 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:03:16.902109    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e357d7_757d_4f89_837e_6283d4f87e11.slice/crio-848b6205eba27f5b2109c4a076155d3ff22c12a7b2936bbbdec11c03ff22acf2 WatchSource:0}: Error finding container 848b6205eba27f5b2109c4a076155d3ff22c12a7b2936bbbdec11c03ff22acf2: Status 404 returned error can't find the container with id 848b6205eba27f5b2109c4a076155d3ff22c12a7b2936bbbdec11c03ff22acf2
Oct 31 13:03:17.172805 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.172699    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-swfl5"]
Oct 31 13:03:17.177064 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.177043    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-swfl5"
Oct 31 13:03:17.177632 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.177613    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:17.188682 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.188661    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-swfl5"]
Oct 31 13:03:17.189094 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.189070    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 13:03:17.189197 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.189071    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 13:03:17.189197 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.189082    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-66zbj\""
Oct 31 13:03:17.256526 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.256492    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f2kwh"]
Oct 31 13:03:17.256839 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.256701    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:17.256839 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.256756    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8s79\" (UniqueName: \"kubernetes.io/projected/d8d147e3-324f-416b-8155-4d3631c5667e-kube-api-access-s8s79\") pod \"migrator-66df5d496b-swfl5\" (UID: \"d8d147e3-324f-416b-8155-4d3631c5667e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-swfl5"
Oct 31 13:03:17.256949 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:17.256860    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 13:03:17.257004 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:17.256949    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls podName:cb046f36-d918-4cdd-b2be-f2eec97ae74e nodeName:}" failed. No retries permitted until 2026-10-31 13:03:21.256929835 +0000 UTC m=+39.167405044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls") pod "dns-default-kstrj" (UID: "cb046f36-d918-4cdd-b2be-f2eec97ae74e") : secret "dns-default-metrics-tls" not found
Oct 31 13:03:17.294472 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.294434    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f2kwh"]
Oct 31 13:03:17.294605 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.294563    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:17.357310 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.357275    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8s79\" (UniqueName: \"kubernetes.io/projected/d8d147e3-324f-416b-8155-4d3631c5667e-kube-api-access-s8s79\") pod \"migrator-66df5d496b-swfl5\" (UID: \"d8d147e3-324f-416b-8155-4d3631c5667e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-swfl5"
Oct 31 13:03:17.381669 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.381643    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8s79\" (UniqueName: \"kubernetes.io/projected/d8d147e3-324f-416b-8155-4d3631c5667e-kube-api-access-s8s79\") pod \"migrator-66df5d496b-swfl5\" (UID: \"d8d147e3-324f-416b-8155-4d3631c5667e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-swfl5"
Oct 31 13:03:17.387405 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.387376    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:17.392193 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.392174    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 13:03:17.392304 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.392215    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tmfz7\""
Oct 31 13:03:17.398209 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.398187    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 13:03:17.398302 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.398247    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 13:03:17.398302 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.398247    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 13:03:17.457838 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.457779    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/852969d3-8684-454c-ac42-56f80877702a-data-volume\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:17.457838 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.457809    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852969d3-8684-454c-ac42-56f80877702a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:17.457838 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.457825    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:17.458043 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.457844    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t24pv\" (UniqueName: \"kubernetes.io/projected/852969d3-8684-454c-ac42-56f80877702a-kube-api-access-t24pv\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:17.458043 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.457998    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/852969d3-8684-454c-ac42-56f80877702a-crio-socket\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:17.507276 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.507246    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-swfl5"
Oct 31 13:03:17.543252 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.543222    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:17.559285 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.559256    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/852969d3-8684-454c-ac42-56f80877702a-crio-socket\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:17.559408 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.559330    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/852969d3-8684-454c-ac42-56f80877702a-data-volume\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:17.559408 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.559356    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852969d3-8684-454c-ac42-56f80877702a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:17.559408 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.559379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:17.559550 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.559407    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t24pv\" (UniqueName: \"kubernetes.io/projected/852969d3-8684-454c-ac42-56f80877702a-kube-api-access-t24pv\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:17.559959 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.559937    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/852969d3-8684-454c-ac42-56f80877702a-crio-socket\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:17.560277 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.560257    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/852969d3-8684-454c-ac42-56f80877702a-data-volume\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:17.560358 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:17.560342    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 13:03:17.560422 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:17.560412    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/852969d3-8684-454c-ac42-56f80877702a-kube-rbac-proxy-cm podName:852969d3-8684-454c-ac42-56f80877702a nodeName:}" failed. No retries permitted until 2026-10-31 13:03:18.060393152 +0000 UTC m=+35.970868365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/852969d3-8684-454c-ac42-56f80877702a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2kwh" (UID: "852969d3-8684-454c-ac42-56f80877702a") : configmap "kube-rbac-proxy" not found
Oct 31 13:03:17.560677 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:17.560661    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:17.560757 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:17.560711    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls podName:852969d3-8684-454c-ac42-56f80877702a nodeName:}" failed. No retries permitted until 2026-10-31 13:03:18.06069782 +0000 UTC m=+35.971173035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2kwh" (UID: "852969d3-8684-454c-ac42-56f80877702a") : secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:17.573660 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.573637    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t24pv\" (UniqueName: \"kubernetes.io/projected/852969d3-8684-454c-ac42-56f80877702a-kube-api-access-t24pv\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:17.817120 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.817090    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-swfl5"]
Oct 31 13:03:17.822773 ip-10-0-136-168 kubenswrapper[2575]: W1031 13:03:17.822737    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d147e3_324f_416b_8155_4d3631c5667e.slice/crio-35752626d455e1a398e311d6ccee74ca2224f51b3b79a71d8b5861da2a1ffaaf WatchSource:0}: Error finding container 35752626d455e1a398e311d6ccee74ca2224f51b3b79a71d8b5861da2a1ffaaf: Status 404 returned error can't find the container with id 35752626d455e1a398e311d6ccee74ca2224f51b3b79a71d8b5861da2a1ffaaf
Oct 31 13:03:17.850669 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.850640    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xpskx" event={"ID":"12e357d7-757d-4f89-837e-6283d4f87e11","Type":"ContainerStarted","Data":"848b6205eba27f5b2109c4a076155d3ff22c12a7b2936bbbdec11c03ff22acf2"}
Oct 31 13:03:17.852149 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:17.852118    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-swfl5" event={"ID":"d8d147e3-324f-416b-8155-4d3631c5667e","Type":"ContainerStarted","Data":"35752626d455e1a398e311d6ccee74ca2224f51b3b79a71d8b5861da2a1ffaaf"}
Oct 31 13:03:18.063674 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:18.063643    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852969d3-8684-454c-ac42-56f80877702a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:18.063839 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:18.063683    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:18.063839 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:18.063775    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 13:03:18.063839 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:18.063829    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:18.064006 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:18.063832    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/852969d3-8684-454c-ac42-56f80877702a-kube-rbac-proxy-cm podName:852969d3-8684-454c-ac42-56f80877702a nodeName:}" failed. No retries permitted until 2026-10-31 13:03:19.063818477 +0000 UTC m=+36.974293672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/852969d3-8684-454c-ac42-56f80877702a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2kwh" (UID: "852969d3-8684-454c-ac42-56f80877702a") : configmap "kube-rbac-proxy" not found
Oct 31 13:03:18.064006 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:18.063943    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls podName:852969d3-8684-454c-ac42-56f80877702a nodeName:}" failed. No retries permitted until 2026-10-31 13:03:19.063901794 +0000 UTC m=+36.974377012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2kwh" (UID: "852969d3-8684-454c-ac42-56f80877702a") : secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:18.190276 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:18.190199    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:19.072491 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:19.072453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852969d3-8684-454c-ac42-56f80877702a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:19.072962 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:19.072502    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:19.072962 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:19.072663    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:19.072962 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:19.072725    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls podName:852969d3-8684-454c-ac42-56f80877702a nodeName:}" failed. No retries permitted until 2026-10-31 13:03:21.072708028 +0000 UTC m=+38.983183237 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2kwh" (UID: "852969d3-8684-454c-ac42-56f80877702a") : secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:19.073161 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:19.073123    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 13:03:19.073215 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:19.073179    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/852969d3-8684-454c-ac42-56f80877702a-kube-rbac-proxy-cm podName:852969d3-8684-454c-ac42-56f80877702a nodeName:}" failed. No retries permitted until 2026-10-31 13:03:21.073162523 +0000 UTC m=+38.983637733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/852969d3-8684-454c-ac42-56f80877702a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2kwh" (UID: "852969d3-8684-454c-ac42-56f80877702a") : configmap "kube-rbac-proxy" not found
Oct 31 13:03:19.354923 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:19.354828    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:20.855399 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:20.855365    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:20.863075 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:20.863044    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jnzs8" event={"ID":"575eac51-2cde-4d41-9de7-f6987fb83dfe","Type":"ContainerStarted","Data":"16581c0558e68ad4231642c9784a115d07b4b1afb491167c2c1601dea84d3113"}
Oct 31 13:03:20.865354 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:20.865329    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-swfl5" event={"ID":"d8d147e3-324f-416b-8155-4d3631c5667e","Type":"ContainerStarted","Data":"3f4f73bc6cd3bb1628d730cb7c457956a286b64466d09a63fcac408bead6c3ed"}
Oct 31 13:03:20.884254 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:20.884212    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jnzs8" podStartSLOduration=2.270289802 podStartE2EDuration="7.884201698s" totalImagesPullingTime="5.613911896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:20.884160123 +0000 UTC m=+38.794635357" watchObservedRunningTime="2026-10-31 13:03:20.884201698 +0000 UTC m=+38.794676932"
Oct 31 13:03:21.089370 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:21.089327    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852969d3-8684-454c-ac42-56f80877702a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:21.089370 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:21.089373    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:21.089625 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:21.089506    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:21.089625 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:21.089575    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls podName:852969d3-8684-454c-ac42-56f80877702a nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.089556268 +0000 UTC m=+43.000031489 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2kwh" (UID: "852969d3-8684-454c-ac42-56f80877702a") : secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:21.090006 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:21.089982    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852969d3-8684-454c-ac42-56f80877702a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:21.291518 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:21.291478    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:21.291710 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:21.291653    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 13:03:21.291776 ip-10-0-136-168 kubenswrapper[2575]: E1031 13:03:21.291729    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls podName:cb046f36-d918-4cdd-b2be-f2eec97ae74e nodeName:}" failed. No retries permitted until 2026-10-31 13:03:29.291709839 +0000 UTC m=+47.202185050 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls") pod "dns-default-kstrj" (UID: "cb046f36-d918-4cdd-b2be-f2eec97ae74e") : secret "dns-default-metrics-tls" not found
Oct 31 13:03:22.149816 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:22.149779    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:22.352231 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:22.352197    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:22.528914 ip-10-0-136-168 kubenswrapper[2575]: I1031 13:03:22.528861    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:22.535957 ip-10-0-136-168 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 13:03:22.547399 ip-10-0-136-168 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 13:03:22.547601 ip-10-0-136-168 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 13:03:22.547664 ip-10-0-136-168 systemd[1]: kubelet.service: Consumed 2.232s CPU time.
Oct 31 13:03:22.559505 ip-10-0-136-168 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 13:03:22.660218 ip-10-0-136-168 kubenswrapper[4428]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 13:03:22.660218 ip-10-0-136-168 kubenswrapper[4428]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 13:03:22.660218 ip-10-0-136-168 kubenswrapper[4428]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 13:03:22.660218 ip-10-0-136-168 kubenswrapper[4428]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 13:03:22.664262 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664153    4428 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:03:22.664262 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664259    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:03:22.664262 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664265    4428 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:03:22.664262 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664270    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664274    4428 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664281    4428 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664285    4428 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664289    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664292    4428 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664296    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664301    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664305    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664309    4428 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664312    4428 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664316    4428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664320    4428 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664324    4428 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664328    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664332    4428 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664336    4428 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664341    4428 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664346    4428 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664351    4428 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:03:22.664520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664355    4428 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664359    4428 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664364    4428 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664369    4428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664374    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664378    4428 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664383    4428 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664388    4428 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664394    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664398    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664403    4428 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664407    4428 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664413    4428 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664417    4428 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664421    4428 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664428    4428 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664432    4428 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664437    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664441    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664446    4428 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:03:22.665222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664449    4428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664451    4428 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664454    4428 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664457    4428 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664459    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664462    4428 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664464    4428 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664468    4428 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664471    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664474    4428 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664476    4428 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664479    4428 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664481    4428 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664485    4428 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664489    4428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664494    4428 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664498    4428 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664503    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664506    4428 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:03:22.665823 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664510    4428 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664516    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664522    4428 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664527    4428 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664532    4428 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664537    4428 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664542    4428 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664546    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664549    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664554    4428 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664558    4428 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664562    4428 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664566    4428 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664569    4428 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664574    4428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664578    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664582    4428 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664586    4428 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664590    4428 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:03:22.666520 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664594    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664598    4428 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664605    4428 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664609    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664613    4428 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664618    4428 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664622    4428 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664627    4428 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664632    4428 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664637    4428 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664641    4428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664646    4428 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664650    4428 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664655    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664660    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664664    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664669    4428 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664676    4428 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664683    4428 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:03:22.667209 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664688    4428 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664694    4428 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664699    4428 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664704    4428 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664710    4428 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.664716    4428 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665577    4428 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665588    4428 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665593    4428 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665599    4428 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665603    4428 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665608    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665613    4428 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665617    4428 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665621    4428 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665626    4428 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665631    4428 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665637    4428 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665642    4428 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:03:22.667930 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665646    4428 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665651    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665655    4428 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665658    4428 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665662    4428 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665666    4428 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665670    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665675    4428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665679    4428 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665682    4428 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665686    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665690    4428 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665695    4428 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665698    4428 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665703    4428 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665706    4428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665710    4428 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665715    4428 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665717    4428 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:03:22.668622 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665720    4428 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665722    4428 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665725    4428 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665727    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665730    4428 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665734    4428 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665739    4428 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665744    4428 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665749    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665754    4428 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665759    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665769    4428 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665774    4428 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665777    4428 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665781    4428 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665785    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665789    4428 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665794    4428 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665799    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665804    4428 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:03:22.669364 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665809    4428 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665813    4428 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665818    4428 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665822    4428 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665826    4428 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665830    4428 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665834    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665838    4428 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665842    4428 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665846    4428 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665850    4428 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665854    4428 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665857    4428 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665860    4428 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665878    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665883    4428 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665889    4428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665893    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665896    4428 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:03:22.670120 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665900    4428 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665903    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665907    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665911    4428 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665915    4428 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665918    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665921    4428 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665923    4428 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665926    4428 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665929    4428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665931    4428 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665934    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665937    4428 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665939    4428 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665941    4428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665944    4428 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665946    4428 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665949    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665951    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665953    4428 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:03:22.670884 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665957    4428 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665962    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665966    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665971    4428 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665975    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665979    4428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665984    4428 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665990    4428 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.665998    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666002    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666007    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666011    4428 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666015    4428 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666019    4428 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666031    4428 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666722    4428 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666733    4428 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666737    4428 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666740    4428 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:03:22.671577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666744    4428 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666747    4428 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666750    4428 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666752    4428 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666755    4428 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666758    4428 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666760    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666763    4428 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666766    4428 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666769    4428 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666771    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666774    4428 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666776    4428 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666780    4428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666785    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666789    4428 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666793    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666797    4428 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666800    4428 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666803    4428 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:03:22.672273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666805    4428 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666807    4428 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666810    4428 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666813    4428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666815    4428 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666817    4428 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666820    4428 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666823    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666826    4428 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666830    4428 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666835    4428 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666840    4428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666845    4428 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666849    4428 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666854    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666859    4428 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666878    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666883    4428 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666888    4428 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:03:22.672818 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666893    4428 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666897    4428 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666901    4428 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666905    4428 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666907    4428 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666910    4428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666913    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666915    4428 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666917    4428 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666920    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666922    4428 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666925    4428 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666928    4428 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666930    4428 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666933    4428 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666935    4428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666938    4428 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666941    4428 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666943    4428 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:03:22.673507 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666946    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666949    4428 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666951    4428 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666955    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666958    4428 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666964    4428 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666969    4428 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666973    4428 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666978    4428 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666982    4428 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666986    4428 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666990    4428 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666994    4428 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.666998    4428 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667002    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667006    4428 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667011    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667014    4428 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667017    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667022    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:03:22.674185 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667026    4428 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667030    4428 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667033    4428 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667037    4428 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667042    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667046    4428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667049    4428 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667053    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667059    4428 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667064    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667068    4428 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667070    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667076    4428 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667081    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667088    4428 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667094    4428 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667098    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667102    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667106    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:03:22.674921 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667111    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667115    4428 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667119    4428 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667123    4428 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.667127    4428 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667253    4428 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667275    4428 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667285    4428 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667292    4428 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667298    4428 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667303    4428 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667310    4428 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667317    4428 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667322    4428 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667333    4428 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667339    4428 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667345    4428 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667350    4428 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667355    4428 flags.go:64] FLAG: --cgroup-root=""
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667359    4428 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667365    4428 flags.go:64] FLAG: --client-ca-file=""
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667370    4428 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667378    4428 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667384    4428 flags.go:64] FLAG: --cluster-domain=""
Oct 31 13:03:22.675637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667389    4428 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667395    4428 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667401    4428 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667407    4428 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667421    4428 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667426    4428 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667431    4428 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667437    4428 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667442    4428 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667450    4428 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667455    4428 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667460    4428 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667464    4428 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667471    4428 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667475    4428 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667480    4428 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667484    4428 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667488    4428 flags.go:64] FLAG: --enable-server="true"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667492    4428 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667499    4428 flags.go:64] FLAG: --event-burst="100"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667505    4428 flags.go:64] FLAG: --event-qps="50"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667509    4428 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667514    4428 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667519    4428 flags.go:64] FLAG: --eviction-hard=""
Oct 31 13:03:22.676581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667526    4428 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667531    4428 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667537    4428 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667542    4428 flags.go:64] FLAG: --eviction-soft=""
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667547    4428 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667552    4428 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667560    4428 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667565    4428 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667568    4428 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667572    4428 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667578    4428 flags.go:64] FLAG: --feature-gates=""
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667584    4428 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667590    4428 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667595    4428 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667600    4428 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667604    4428 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667609    4428 flags.go:64] FLAG: --help="false"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667613    4428 flags.go:64] FLAG: --hostname-override="ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667620    4428 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667624    4428 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667627    4428 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667630    4428 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667635    4428 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 13:03:22.677764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667640    4428 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667644    4428 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667649    4428 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667654    4428 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667659    4428 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667665    4428 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667670    4428 flags.go:64] FLAG: --kube-reserved=""
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667674    4428 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667677    4428 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667680    4428 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667683    4428 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667686    4428 flags.go:64] FLAG: --lock-file=""
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667688    4428 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667691    4428 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667695    4428 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667706    4428 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667711    4428 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667716    4428 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667721    4428 flags.go:64] FLAG: --logging-format="text"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667727    4428 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667732    4428 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667738    4428 flags.go:64] FLAG: --manifest-url=""
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667741    4428 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667745    4428 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667749    4428 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 13:03:22.678640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667753    4428 flags.go:64] FLAG: --max-pods="110"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667756    4428 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667759    4428 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667763    4428 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667768    4428 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667772    4428 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667777    4428 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667782    4428 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667793    4428 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667798    4428 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667802    4428 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667807    4428 flags.go:64] FLAG: --pod-cidr=""
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667811    4428 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667814    4428 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667817    4428 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667820    4428 flags.go:64] FLAG: --port="10250"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667824    4428 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667829    4428 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-09fbf5151f39a4afe"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667835    4428 flags.go:64] FLAG: --qos-reserved=""
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667840    4428 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667845    4428 flags.go:64] FLAG: --register-node="true"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667850    4428 flags.go:64] FLAG: --register-with-taints=""
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667858    4428 flags.go:64] FLAG: --registry-burst="10"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667879    4428 flags.go:64] FLAG: --registry-qps="5"
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667884    4428 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667890    4428 flags.go:64] FLAG: --reserved-memory=""
Oct 31 13:03:22.679750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667897    4428 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667902    4428 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667908    4428 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667912    4428 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667916    4428 flags.go:64] FLAG: --runonce="false"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667920    4428 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667925    4428 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667930    4428 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667935    4428 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667940    4428 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667947    4428 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667952    4428 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667957    4428 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667962    4428 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667967    4428 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667972    4428 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667977    4428 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667982    4428 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667986    4428 flags.go:64] FLAG: --system-cgroups=""
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667989    4428 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667995    4428 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.667998    4428 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.668001    4428 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.668008    4428 flags.go:64] FLAG: --tls-min-version=""
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.668012    4428 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 13:03:22.680712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.668016    4428 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.668021    4428 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.668026    4428 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.668033    4428 flags.go:64] FLAG: --v="2"
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.668046    4428 flags.go:64] FLAG: --version="false"
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.668052    4428 flags.go:64] FLAG: --vmodule=""
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.668057    4428 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.668063    4428 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668274    4428 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668282    4428 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668287    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668291    4428 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668295    4428 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668299    4428 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668304    4428 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668309    4428 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668315    4428 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668321    4428 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668326    4428 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668329    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668331    4428 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:03:22.681776 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668335    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668339    4428 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668344    4428 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668347    4428 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668352    4428 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668356    4428 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668359    4428 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668363    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668367    4428 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668371    4428 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668374    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668379    4428 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668383    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668387    4428 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668394    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668399    4428 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668404    4428 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668408    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668412    4428 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668416    4428 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:03:22.682614 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668421    4428 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668425    4428 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668429    4428 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668432    4428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668436    4428 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668440    4428 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668444    4428 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668448    4428 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668451    4428 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668456    4428 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668461    4428 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668464    4428 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668467    4428 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668469    4428 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668473    4428 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668479    4428 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668486    4428 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668491    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668497    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:03:22.683581 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668501    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668505    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668509    4428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668513    4428 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668517    4428 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668521    4428 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668523    4428 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668529    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668533    4428 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668537    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668541    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668545    4428 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668548    4428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668553    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668557    4428 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668561    4428 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668565    4428 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668569    4428 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668573    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668577    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:03:22.684360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668582    4428 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668585    4428 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668588    4428 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668590    4428 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668595    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668599    4428 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668603    4428 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668606    4428 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668610    4428 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668614    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668618    4428 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668622    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668626    4428 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668630    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668635    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668640    4428 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668644    4428 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668648    4428 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668652    4428 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:03:22.685183 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668657    4428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:03:22.685934 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668662    4428 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:03:22.685934 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668666    4428 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:03:22.685934 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668670    4428 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:03:22.685934 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668674    4428 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:03:22.685934 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668677    4428 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:03:22.685934 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668681    4428 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:03:22.685934 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668685    4428 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:03:22.685934 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668689    4428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:03:22.685934 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668693    4428 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:03:22.685934 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668698    4428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:03:22.685934 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668701    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:03:22.685934 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668703    4428 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:03:22.685934 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668706    4428 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:03:22.685934 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668708    4428 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:03:22.685934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.668752    4428 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 13:03:22.685934 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668806    4428 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668815    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668820    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668825    4428 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668829    4428 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668833    4428 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668837    4428 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668843    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668847    4428 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668851    4428 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668855    4428 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668859    4428 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668878    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668883    4428 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668887    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668891    4428 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668897    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668901    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668905    4428 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:03:22.686491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668908    4428 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668913    4428 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668917    4428 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668920    4428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668923    4428 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668928    4428 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668933    4428 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668937    4428 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668941    4428 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668945    4428 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668949    4428 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668953    4428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668957    4428 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668961    4428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668965    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668969    4428 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668974    4428 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668978    4428 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668982    4428 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:03:22.687186 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668985    4428 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668989    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668991    4428 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668994    4428 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668996    4428 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.668999    4428 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669001    4428 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669007    4428 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669012    4428 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669017    4428 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669023    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669028    4428 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669032    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669036    4428 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669040    4428 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669044    4428 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669049    4428 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669053    4428 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669057    4428 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669061    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:03:22.687937 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669088    4428 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669095    4428 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669100    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669104    4428 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669108    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669112    4428 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669117    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669120    4428 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669124    4428 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669128    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669133    4428 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669138    4428 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669142    4428 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669148    4428 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669153    4428 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669157    4428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669161    4428 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669166    4428 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669170    4428 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669174    4428 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:03:22.688730 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669178    4428 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669182    4428 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669187    4428 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669191    4428 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669196    4428 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669199    4428 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669204    4428 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669210    4428 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669214    4428 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669217    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669221    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669226    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669230    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669235    4428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669240    4428 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669250    4428 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669254    4428 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669259    4428 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669263    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:03:22.689340 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669266    4428 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669270    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669274    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669278    4428 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669283    4428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669287    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669292    4428 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669294    4428 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.669297    4428 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.674384    4428 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.674400    4428 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.674418    4428 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.674423    4428 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674498    4428 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674506    4428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674512    4428 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674516    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674520    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674524    4428 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:03:22.689813 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674528    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674533    4428 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674537    4428 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674540    4428 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674542    4428 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674545    4428 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674548    4428 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674552    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674556    4428 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674561    4428 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674565    4428 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674570    4428 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674574    4428 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674578    4428 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674581    4428 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674585    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674589    4428 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674593    4428 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674597    4428 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674601    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:03:22.690386 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674606    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674610    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674613    4428 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674618    4428 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674623    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674628    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674632    4428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674637    4428 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674639    4428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674642    4428 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674644    4428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674647    4428 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674649    4428 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674651    4428 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674654    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674658    4428 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674662    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674665    4428 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674667    4428 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:03:22.690988 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674670    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674672    4428 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674674    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674677    4428 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674680    4428 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674682    4428 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674684    4428 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674687    4428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674691    4428 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674698    4428 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674705    4428 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674710    4428 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674714    4428 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674716    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674719    4428 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674722    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674724    4428 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674728    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674731    4428 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:03:22.691613 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674733    4428 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674736    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674738    4428 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674741    4428 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674743    4428 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674745    4428 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674748    4428 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674750    4428 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674753    4428 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674757    4428 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674762    4428 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674767    4428 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674772    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674776    4428 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674779    4428 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674784    4428 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674789    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674793    4428 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674797    4428 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:03:22.692222 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674802    4428 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674806    4428 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674810    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674814    4428 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674818    4428 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674823    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674827    4428 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674831    4428 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674834    4428 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674838    4428 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674841    4428 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674847    4428 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674853    4428 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674857    4428 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674862    4428 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674882    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674886    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674891    4428 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674895    4428 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674899    4428 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:03:22.692841 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674904    4428 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:03:22.693506 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674908    4428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:03:22.693506 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.674911    4428 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:03:22.693506 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.674948    4428 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 13:03:22.693506 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675147    4428 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:03:22.693506 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675156    4428 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:03:22.693506 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675160    4428 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:03:22.693506 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675165    4428 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:03:22.693506 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675169    4428 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:03:22.693506 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675174    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:03:22.693506 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675179    4428 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:03:22.693506 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675183    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:03:22.693506 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675187    4428 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:03:22.693506 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675192    4428 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:03:22.693506 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675197    4428 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:03:22.693506 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675201    4428 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:03:22.693506 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675205    4428 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675208    4428 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675212    4428 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675217    4428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675221    4428 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675225    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675230    4428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675234    4428 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675238    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675245    4428 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675249    4428 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675254    4428 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675258    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675262    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675267    4428 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675271    4428 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675274    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675278    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675282    4428 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:03:22.693997 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675286    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675290    4428 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675294    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675298    4428 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675302    4428 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675306    4428 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675310    4428 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675314    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675319    4428 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675324    4428 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675327    4428 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675332    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675336    4428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675340    4428 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675344    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675349    4428 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675357    4428 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675362    4428 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675366    4428 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675371    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:03:22.694468 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675375    4428 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675378    4428 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675383    4428 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675388    4428 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675392    4428 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675396    4428 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675400    4428 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675404    4428 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675408    4428 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675412    4428 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675416    4428 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675420    4428 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675423    4428 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675427    4428 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675433    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675437    4428 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675441    4428 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675444    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675446    4428 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675450    4428 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675455    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:03:22.695258 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675459    4428 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675464    4428 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675468    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675474    4428 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675479    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675484    4428 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675488    4428 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675496    4428 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675501    4428 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675508    4428 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675514    4428 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675519    4428 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675524    4428 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675529    4428 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675534    4428 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675538    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675542    4428 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675545    4428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675552    4428 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:03:22.696111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675557    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:03:22.697117 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675562    4428 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:03:22.697117 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675567    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:03:22.697117 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675571    4428 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:03:22.697117 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675576    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:03:22.697117 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675580    4428 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:03:22.697117 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675585    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:03:22.697117 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675589    4428 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:03:22.697117 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675595    4428 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:03:22.697117 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675599    4428 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:03:22.697117 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675602    4428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:03:22.697117 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675605    4428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:03:22.697117 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675607    4428 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:03:22.697117 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675610    4428 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:03:22.697117 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675612    4428 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:03:22.697117 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.675658    4428 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675693    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675702    4428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675707    4428 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675713    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675717    4428 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675723    4428 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675727    4428 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675729    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675732    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675734    4428 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675737    4428 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675739    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675742    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675745    4428 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675748    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675753    4428 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675758    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675762    4428 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675767    4428 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:03:22.697523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675771    4428 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675776    4428 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675780    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675784    4428 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675789    4428 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675793    4428 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675797    4428 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675800    4428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675804    4428 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675809    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675813    4428 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675819    4428 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675823    4428 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675829    4428 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675833    4428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675836    4428 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675839    4428 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675841    4428 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675845    4428 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675848    4428 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:03:22.698060 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675850    4428 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675853    4428 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675855    4428 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675858    4428 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675860    4428 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675881    4428 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675886    4428 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675891    4428 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675896    4428 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675900    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675904    4428 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675909    4428 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675913    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675917    4428 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675921    4428 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675925    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675929    4428 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675932    4428 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675935    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675937    4428 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:03:22.698539 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675940    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675942    4428 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675944    4428 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675947    4428 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675949    4428 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675953    4428 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675957    4428 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675960    4428 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675963    4428 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675967    4428 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675972    4428 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675976    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675979    4428 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675982    4428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675985    4428 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675988    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675990    4428 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675993    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675995    4428 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:03:22.699044 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.675998    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676003    4428 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676006    4428 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676009    4428 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676013    4428 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676016    4428 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676020    4428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676022    4428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676025    4428 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676027    4428 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676030    4428 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676032    4428 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676042    4428 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676047    4428 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676052    4428 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676056    4428 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676059    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676061    4428 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676064    4428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:03:22.699551 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676066    4428 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:03:22.700032 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676069    4428 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:03:22.700032 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676071    4428 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:03:22.700032 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676073    4428 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:03:22.700032 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676077    4428 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:03:22.700032 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676080    4428 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:03:22.700032 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676083    4428 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:03:22.700032 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676086    4428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:03:22.700032 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:22.676088    4428 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:03:22.700032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.676151    4428 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 13:03:22.700032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.677203    4428 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 13:03:22.700032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.677257    4428 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 13:03:22.700032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.677442    4428 server.go:1014] "Starting client certificate rotation"
Oct 31 13:03:22.700032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.677601    4428 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 13:03:22.700032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.677668    4428 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:57:42 +0000 UTC" deadline="2028-05-04 03:59:47.333648594 +0000 UTC"
Oct 31 13:03:22.700032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.677705    4428 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13214h56m24.655946609s"
Oct 31 13:03:22.700032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.678046    4428 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 13:03:22.700625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.679596    4428 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 13:03:22.700625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.683680    4428 log.go:25] "Validated CRI v1 runtime API"
Oct 31 13:03:22.700625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.685718    4428 log.go:25] "Validated CRI v1 image API"
Oct 31 13:03:22.700625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.686700    4428 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 13:03:22.700625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.688451    4428 fs.go:135] Filesystem UUIDs: map[157a36a1-2b0e-4f40-bb36-d5a3a8aa8c83:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 eb16f3e7-25bd-4f05-91d4-89ab7201b35a:/dev/nvme0n1p3]
Oct 31 13:03:22.700956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.688466    4428 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/093df186420655fb58f0dd5c6e8b6da524689a7306f29de04d4e28fd8f401f0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/093df186420655fb58f0dd5c6e8b6da524689a7306f29de04d4e28fd8f401f0b/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/213c756f5bca005218873d47904eb58eefc7370d0138aa71a05d3fe97027d031/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/213c756f5bca005218873d47904eb58eefc7370d0138aa71a05d3fe97027d031/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30d4e2189000346936788ad998ca7abd115d9f270de12e6ec7ffa0e33f0ae607/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30d4e2189000346936788ad998ca7abd115d9f270de12e6ec7ffa0e33f0ae607/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35752626d455e1a398e311d6ccee74ca2224f51b3b79a71d8b5861da2a1ffaaf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35752626d455e1a398e311d6ccee74ca2224f51b3b79a71d8b5861da2a1ffaaf/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f3028dabf28d8ee66781a6f70ab02c40f38e5d1ab4040ab5c6c7e65b8925059/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f3028dabf28d8ee66781a6f70ab02c40f38e5d1ab4040ab5c6c7e65b8925059/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c94c9269862aa3d9eee1072f8aa8c5c494e7de5d325a8b254e31eafc07eefc0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c94c9269862aa3d9eee1072f8aa8c5c494e7de5d325a8b254e31eafc07eefc0/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f66ffa1bb68611960cb1b62e73cde35ee058af9bf249afb39a19dee9fedba87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f66ffa1bb68611960cb1b62e73cde35ee058af9bf249afb39a19dee9fedba87/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/848b6205eba27f5b2109c4a076155d3ff22c12a7b2936bbbdec11c03ff22acf2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/848b6205eba27f5b2109c4a076155d3ff22c12a7b2936bbbdec11c03ff22acf2/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a553509e10e565e44f362d84264a307b7baea6af3150ae2386065e0ebba4882/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a553509e10e565e44f362d84264a307b7baea6af3150ae2386065e0ebba4882/userdata/shm major:0 minor:71 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be87dc68656727a978cb575eb654c681df63fe90007912a1d65ee06435c10d95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be87dc68656727a978cb575eb654c681df63fe90007912a1d65ee06435c10d95/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c82bdb7624c5c7e1724272d83a313273acd800ceae755fe5179ccfab6d9d1ff8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c82bdb7624c5c7e1724272d83a313273acd800ceae755fe5179ccfab6d9d1ff8/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0007b66a1deb53d8b3459e5c7a8d692ea642205e62878bc20b1f25b1955d151/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0007b66a1deb53d8b3459e5c7a8d692ea642205e62878bc20b1f25b1955d151/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db3a7590fe5683b9717d7a2608a170530c7b18c2e42a2fabc409c9e290c0e357/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db3a7590fe5683b9717d7a2608a170530c7b18c2e42a2fabc409c9e290c0e357/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/edb3fe01cecc1e7d776497d6eb9af7760bb04ded5fe7de53cf0d97e7d547fda0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/edb3fe01cecc1e7d776497d6eb9af7760bb04ded5fe7de53cf0d97e7d547fda0/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0edfba66-6a3a-4be8-b420-227d0443ddea/volumes/kubernetes.io~projected/kube-api-access-zhrzc:{mountpoint:/var/lib/kubelet/pods/0edfba66-6a3a-4be8-b420-227d0443ddea/volumes/kubernetes.io~projected/kube-api-access-zhrzc major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/103c4c86-a9f6-4a59-a615-940717e39dc9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/103c4c86-a9f6-4a59-a615-940717e39dc9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/103c4c86-a9f6-4a59-a615-940717e39dc9/volumes/kubernetes.io~projected/kube-api-access-f7cnz:{mountpoint:/var/lib/kubelet/pods/103c4c86-a9f6-4a59-a615-940717e39dc9/volumes/kubernetes.io~projected/kube-api-access-f7cnz major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/103c4c86-a9f6-4a59-a615-940717e39dc9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/103c4c86-a9f6-4a59-a615-940717e39dc9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12e357d7-757d-4f89-837e-6283d4f87e11/volumes/kubernetes.io~projected/kube-api-access-kb2z7:{mountpoint:/var/lib/kubelet/pods/12e357d7-757d-4f89-837e-6283d4f87e11/volumes/kubernetes.io~projected/kube-api-access-kb2z7 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f97d160-4ad4-4546-9310-4a20d1309648/volumes/kubernetes.io~projected/kube-api-access-82sb9:{mountpoint:/var/lib/kubelet/pods/2f97d160-4ad4-4546-9310-4a20d1309648/volumes/kubernetes.io~projected/kube-api-access-82sb9 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/575eac51-2cde-4d41-9de7-f6987fb83dfe/volumes/kubernetes.io~projected/kube-api-access-64rv7:{mountpoint:/var/lib/kubelet/pods/575eac51-2cde-4d41-9de7-f6987fb83dfe/volumes/kubernetes.io~projected/kube-api-access-64rv7 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/575eac51-2cde-4d41-9de7-f6987fb83dfe/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/575eac51-2cde-4d41-9de7-f6987fb83dfe/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46/volumes/kubernetes.io~projected/kube-api-access-5xl4l:{mountpoint:/var/lib/kubelet/pods/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46/volumes/kubernetes.io~projected/kube-api-access-5xl4l major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62aae01d-fd1d-43c3-9ad7-965bc6d595e2/volumes/kubernetes.io~projected/kube-api-access-s5hcb:{mountpoint:/var/lib/kubelet/pods/62aae01d-fd1d-43c3-9ad7-965bc6d595e2/volumes/kubernetes.io~projected/kube-api-access-s5hcb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d/volumes/kubernetes.io~projected/kube-api-access-94j67:{mountpoint:/var/lib/kubelet/pods/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d/volumes/kubernetes.io~projected/kube-api-access-94j67 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82689b74-1468-4087-8123-b361839b4124/volumes/kubernetes.io~projected/kube-api-access-tdl5n:{mountpoint:/var/lib/kubelet/pods/82689b74-1468-4087-8123-b361839b4124/volumes/kubernetes.io~projected/kube-api-access-tdl5n major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/852969d3-8684-454c-ac42-56f80877702a/volumes/kubernetes.io~projected/kube-api-access-t24pv:{mountpoint:/var/lib/kubelet/pods/852969d3-8684-454c-ac42-56f80877702a/volumes/kubernetes.io~projected/kube-api-access-t24pv major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d617e36-a72a-4a8c-bc9c-85500f109e42/volumes/kubernetes.io~projected/kube-api-access-bmbrx:{mountpoint:/var/lib/kubelet/pods/8d617e36-a72a-4a8c-bc9c-85500f109e42/volumes/kubernetes.io~projected/kube-api-access-bmbrx major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1b19e39-5b5d-46f7-b005-a1e5165fe135/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c1b19e39-5b5d-46f7-b005-a1e5165fe135/volumes/kubernetes.io~secret/agent-certs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb046f36-d918-4cdd-b2be-f2eec97ae74e/volumes/kubernetes.io~projected/kube-api-access-9rnl5:{mountpoint:/var/lib/kubelet/pods/cb046f36-d918-4cdd-b2be-f2eec97ae74e/volumes/kubernetes.io~projected/kube-api-access-9rnl5 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7a89a36-092a-4d38-8a50-60619cc50832/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d7a89a36-092a-4d38-8a50-60619cc50832/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8d147e3-324f-416b-8155-4d3631c5667e/volumes/kubernetes.io~projected/kube-api-access-s8s79:{mountpoint:/var/lib/kubelet/pods/d8d147e3-324f-416b-8155-4d3631c5667e/volumes/kubernetes.io~projected/kube-api-access-s8s79 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f24ca8d4-b321-4579-aff7-e1d0177f50bc/volumes/kubernetes.io~projected/kube-api-access-m78mh:{mountpoint:/var/lib/kubelet/pods/f24ca8d4-b321-4579-aff7-e1d0177f50bc/volumes/kubernetes.io~projected/kube-api-access-m78mh major:0 minor:60 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f9cf0fb501f191e9043c2e60e2ea94f8a8cded30b49715d1f84d6bd0f2a102e1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/1070073965e8936e22ade1d738f03917a73def3806641581b994f65688991c32/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d01bf1ef65d84cbb3233c27db2df31220347458f22f81ff3945cd0fba71d7725/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7f19ffe2d9971b5daae669801d6099a3acd3abfe3a18e7b53d20accac9e2eb7c/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/96a0ddf3a9d398962f0ceb62fbaa5fbf4c887dc10914d1eee5231d62f0460804/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4f87efd34c6d05c922024bb60eb5bc85a984344b7959336cf9ed22df261d7cc3/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/c04c33ec62b541557b2e428aef96984e005a7a658fa54aff4782d6db456e9ed9/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/7047111f557c32382d206805f556abb8f7fb93ea43f4ca52ea42f143654871ca/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/86cc1183ae23c2b7c2995248aac85678afe2d96d3214c28251058da8babf3ba7/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/faf7802ede36662db150ac62e9683f74a9da77eac62238760cc56402eb6eb289/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/2070559f02c3629db8cb8fdbc0ef209762a4d3bc92701cf6f5829c750cae2c3e/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/40b47de736ca9ff0c879ea5e2599b198ef514b44f88743e0d96aa69c6f16af90/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/9aa4af5aa8267b9a921df2674cc56d918e9b20b7dc8daa1d233770325bd60466/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/394e294192976a41d94f4f9c0f4fd1214baa093ce7a58f3886d9534be64efe3c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/58a4c785e089d6f7a1084e02b9e6ee55cda9d85dadd1d204ecb444c140fa0a15/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d480a300b5f28ebe80db9154801cf3c8c1193fb6ba800f4e47c6f8287377d7c4/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4110671a2a3b5196d4b004cc6da89f2474a0b77fc01f721e4eae5cfda131b623/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/cc0880736b8986a0d4193729d2c01d78fb14f9ce746a32ff9a603a98a26ad3d1/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/7a7afd6f7c78230e8a515a21ad66b87975d767ae376d73ca6c17012eec47fd8a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/5233539434eead09753cbc7f92bea66833db7cc16ac6d89a98bc09873ff99f83/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e4e39e636bc42518119eb5b4eabe1694661c5345e059b6d0748009dd7fd30a1b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/cf6618b5e8a673ba2d02adc619d1d1b025263c43a0699d3096871cc8d149a9cd/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/a58510a37e476e69d4522e8bc89db210a7699cfebdbb2beccbf8dc4673d514b9/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/169d30910c6b14037a5d0d863ba0809f1c5990fd9923324421be2d9cfd19a275/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/22a74934ad2e930335ec51f43201c5a3a0b04c181bdeb1f49a3f4570ebd64dd7/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/930420628fa4cb5520a43d19d73544285e4bed52f59c684d61a232a16ebbf29e/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/2adc543f498e210ce0aea393ade1501dbca8a8208520bd821ca916499436fc10/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/f800722bb583e51956f93c90f7e4b2c262a05a2e293c6d64167d5d5b4fd04cd7/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/38291b6e2dc412242ca6e90f1ced4167ed5ddfa7396ec6f807ec5c21ad32d05d/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/1228e609e76d9f404e62c9761f52ffc5b0b9fe80eadaa8f93b726497a9824690/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-69:{mountpoint:/var/lib/containers/storage/overlay/02c5fcd6da8780a7cb5a2f1a4999e63b254c8c84be9df777ce47abc7edfa92c4/merged major:0 minor:69 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a56e9764dad45d1cc39e7949bea93ea8695fb6213f10b1b49724e669b3188274/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/be74fa735f3e9e1dbf2add3505975b35db052ec230c2f1e97afb19f7702c0713/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/9bd5b8b82fb83f687362e42114328b30a2103c9f114cf9dbbf61b75a914a34f4/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/da9539028c94ad1792eec489333b99da632601a189177f174288fa78a950c730/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/aab2b015e5579046676f1e43be309d3737a00a68acafd81892cb1cad8bb76c10/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/4baee70ccc53bb9fc12f3b100d015e2a2e535a439ea1aa29923f42a60cebffdf/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/9b1d9174c5d1f87f6c8125cb2fe4c5013d6adcf38f1e04600832cd717b7dd048/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.694285    4428 manager.go:217] Machine: {Timestamp:2026-10-31 13:03:22.693809157 +0000 UTC m=+0.092157782 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101307 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aec95a5185cd44cb52f63895644d2 SystemUUID:ec2aec95-a518-5cd4-4cb5-2f63895644d2 BootID:32d5510d-fce8-4e49-9640-e81e3b8edf80 Filesystems:[{Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8d617e36-a72a-4a8c-bc9c-85500f109e42/volumes/kubernetes.io~projected/kube-api-access-bmbrx DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db3a7590fe5683b9717d7a2608a170530c7b18c2e42a2fabc409c9e290c0e357/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/82689b74-1468-4087-8123-b361839b4124/volumes/kubernetes.io~projected/kube-api-access-tdl5n DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f97d160-4ad4-4546-9310-4a20d1309648/volumes/kubernetes.io~projected/kube-api-access-82sb9 DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/edb3fe01cecc1e7d776497d6eb9af7760bb04ded5fe7de53cf0d97e7d547fda0/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d8d147e3-324f-416b-8155-4d3631c5667e/volumes/kubernetes.io~projected/kube-api-access-s8s79 DeviceMajor:0 DeviceMinor:216 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/852969d3-8684-454c-ac42-56f80877702a/volumes/kubernetes.io~projected/kube-api-access-t24pv DeviceMajor:0 DeviceMinor:219 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/62aae01d-fd1d-43c3-9ad7-965bc6d595e2/volumes/kubernetes.io~projected/kube-api-access-s5hcb DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f3028dabf28d8ee66781a6f70ab02c40f38e5d1ab4040ab5c6c7e65b8925059/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30d4e2189000346936788ad998ca7abd115d9f270de12e6ec7ffa0e33f0ae607/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be87dc68656727a978cb575eb654c681df63fe90007912a1d65ee06435c10d95/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/103c4c86-a9f6-4a59-a615-940717e39dc9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f66ffa1bb68611960cb1b62e73cde35ee058af9bf249afb39a19dee9fedba87/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c94c9269862aa3d9eee1072f8aa8c5c494e7de5d325a8b254e31eafc07eefc0/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35752626d455e1a398e311d6ccee74ca2224f51b3b79a71d8b5861da2a1ffaaf/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0007b66a1deb53d8b3459e5c7a8d692ea642205e62878bc20b1f25b1955d151/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0edfba66-6a3a-4be8-b420-227d0443ddea/volumes/kubernetes.io~projected/kube-api-access-zhrzc DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb046f36-d918-4cdd-b2be-f2eec97ae74e/volumes/kubernetes.io~projected/kube-api-access-9rnl5 DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c82bdb7624c5c7e1724272d83a313273acd800ceae755fe5179ccfab6d9d1ff8/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f24ca8d4-b321-4579-aff7-e1d0177f50bc/volumes/kubernetes.io~projected/kube-api-access-m78mh DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a553509e10e565e44f362d84264a307b7baea6af3150ae2386065e0ebba4882/userdata/shm DeviceMajor:0 DeviceMinor:71 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/213c756f5bca005218873d47904eb58eefc7370d0138aa71a05d3fe97027d031/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/103c4c86-a9f6-4a59-a615-940717e39dc9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/575eac51-2cde-4d41-9de7-f6987fb83dfe/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/103c4c86-a9f6-4a59-a615-940717e39dc9/volumes/kubernetes.io~projected/kube-api-access-f7cnz DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7a89a36-092a-4d38-8a50-60619cc50832/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/12e357d7-757d-4f89-837e-6283d4f87e11/volumes/kubernetes.io~projected/kube-api-access-kb2z7 DeviceMajor:0 DeviceMinor:209 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-69 DeviceMajor:0 DeviceMinor:69 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1b19e39-5b5d-46f7-b005-a1e5165fe135/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/848b6205eba27f5b2109c4a076155d3ff22c12a7b2936bbbdec11c03ff22acf2/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/575eac51-2cde-4d41-9de7-f6987fb83dfe/volumes/kubernetes.io~projected/kube-api-access-64rv7 DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d/volumes/kubernetes.io~projected/kube-api-access-94j67 DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46/volumes/kubernetes.io~projected/kube-api-access-5xl4l DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/093df186420655fb58f0dd5c6e8b6da524689a7306f29de04d4e28fd8f401f0b/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:093df186420655f MacAddress:42:46:43:8d:1a:0b Speed:10000 Mtu:8901} {Name:35752626d455e1a MacAddress:7e:55:81:45:79:9d Speed:10000 Mtu:8901} {Name:848b6205eba27f5 MacAddress:6e:3a:63:33:55:8e Speed:10000 Mtu:8901} {Name:be87dc68656727a MacAddress:6e:6e:34:66:7b:8a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f7:4a:7f:67 Speed:0 Mtu:9001} {Name:br-int MacAddress:1a:7a:cb:a3:dc:97 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:f7:4a:7f:67 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:4e:dd:fc:1b:20:03 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1e:f1:80:3a:1a:77 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.694562    4428 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.694614    4428 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.694966    4428 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.694993    4428 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-168.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.695188    4428 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.695201    4428 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.695214    4428 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.695241    4428 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.695362    4428 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.695577    4428 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.695718    4428 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.695768    4428 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.695787    4428 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.695800    4428 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.695815    4428 kubelet.go:405] "Adding apiserver pod source"
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.695826    4428 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.696669    4428 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.696683    4428 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 13:03:22.701194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.697366    4428 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 13:03:22.705493 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.705477    4428 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 13:03:22.705493 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.705491    4428 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 13:03:22.705604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.705590    4428 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 13:03:22.705795 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.705782    4428 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 13:03:22.705826 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.705814    4428 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 13:03:22.705954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.705944    4428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 13:03:22.705986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.705959    4428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 13:03:22.705986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.705965    4428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 13:03:22.705986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.705971    4428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 13:03:22.705986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.705976    4428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 13:03:22.705986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.705982    4428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 13:03:22.705986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.705987    4428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 13:03:22.706164 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.705993    4428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 13:03:22.706164 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.706000    4428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 13:03:22.706164 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.706006    4428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 13:03:22.706164 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.706013    4428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 13:03:22.706164 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.706023    4428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 13:03:22.706164 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.706047    4428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 13:03:22.706164 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.706066    4428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 13:03:22.711007 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.710991    4428 server.go:1263] "Started kubelet"
Oct 31 13:03:22.711157 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.711099    4428 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 13:03:22.711258 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.711178    4428 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 13:03:22.711337 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.711251    4428 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 13:03:22.712123 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.711486    4428 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 13:03:22.711533 ip-10-0-136-168 systemd[1]: Started Kubernetes Kubelet.
Oct 31 13:03:22.712452 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.712438    4428 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 13:03:22.713376 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.713359    4428 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 13:03:22.714696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.713615    4428 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 13:03:22.714696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.713812    4428 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:57:44 +0000 UTC" deadline="2028-05-12 15:23:47.261119011 +0000 UTC"
Oct 31 13:03:22.714696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.713862    4428 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13418h20m24.547260927s"
Oct 31 13:03:22.714696 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:03:22.713946    4428 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-168.us-east-2.compute.internal\" not found"
Oct 31 13:03:22.714696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.714001    4428 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 13:03:22.714696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.714008    4428 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 13:03:22.714696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.714222    4428 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 13:03:22.715350 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.715329    4428 factory.go:55] Registering systemd factory
Oct 31 13:03:22.715455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.715353    4428 factory.go:223] Registration of the systemd container factory successfully
Oct 31 13:03:22.715600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.715587    4428 factory.go:153] Registering CRI-O factory
Oct 31 13:03:22.715684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.715604    4428 factory.go:223] Registration of the crio container factory successfully
Oct 31 13:03:22.715684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.715665    4428 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 13:03:22.715809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.715702    4428 factory.go:103] Registering Raw factory
Oct 31 13:03:22.715809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.715720    4428 manager.go:1199] Started watching for new ooms in manager
Oct 31 13:03:22.715975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.715834    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="852969d3-8684-454c-ac42-56f80877702a" volumeName="kubernetes.io/projected/852969d3-8684-454c-ac42-56f80877702a-kube-api-access-t24pv" seLinuxMountContext=""
Oct 31 13:03:22.715975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.715891    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d617e36-a72a-4a8c-bc9c-85500f109e42" volumeName="kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 13:03:22.715975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.715912    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d617e36-a72a-4a8c-bc9c-85500f109e42" volumeName="kubernetes.io/projected/8d617e36-a72a-4a8c-bc9c-85500f109e42-kube-api-access-bmbrx" seLinuxMountContext=""
Oct 31 13:03:22.715975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.715926    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f1fbe3-7baa-4d79-b758-5c5cdc99f45d" volumeName="kubernetes.io/projected/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-kube-api-access-94j67" seLinuxMountContext=""
Oct 31 13:03:22.715975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.715941    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="852969d3-8684-454c-ac42-56f80877702a" volumeName="kubernetes.io/empty-dir/852969d3-8684-454c-ac42-56f80877702a-data-volume" seLinuxMountContext=""
Oct 31 13:03:22.715975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.715954    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d617e36-a72a-4a8c-bc9c-85500f109e42" volumeName="kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-cni-binary-copy" seLinuxMountContext=""
Oct 31 13:03:22.715975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.715966    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="103c4c86-a9f6-4a59-a615-940717e39dc9" volumeName="kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-env-overrides" seLinuxMountContext=""
Oct 31 13:03:22.715975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.715977    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="103c4c86-a9f6-4a59-a615-940717e39dc9" volumeName="kubernetes.io/projected/103c4c86-a9f6-4a59-a615-940717e39dc9-kube-api-access-f7cnz" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.715995    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f97d160-4ad4-4546-9310-4a20d1309648" volumeName="kubernetes.io/projected/2f97d160-4ad4-4546-9310-4a20d1309648-kube-api-access-82sb9" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716023    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="575eac51-2cde-4d41-9de7-f6987fb83dfe" volumeName="kubernetes.io/secret/575eac51-2cde-4d41-9de7-f6987fb83dfe-cert" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716041    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46" volumeName="kubernetes.io/empty-dir/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-tmp" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716054    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d147e3-324f-416b-8155-4d3631c5667e" volumeName="kubernetes.io/projected/d8d147e3-324f-416b-8155-4d3631c5667e-kube-api-access-s8s79" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716066    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46" volumeName="kubernetes.io/empty-dir/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-tuned" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716097    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46" volumeName="kubernetes.io/projected/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-kube-api-access-5xl4l" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716113    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0edfba66-6a3a-4be8-b420-227d0443ddea" volumeName="kubernetes.io/projected/0edfba66-6a3a-4be8-b420-227d0443ddea-kube-api-access-zhrzc" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716125    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="103c4c86-a9f6-4a59-a615-940717e39dc9" volumeName="kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716137    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="103c4c86-a9f6-4a59-a615-940717e39dc9" volumeName="kubernetes.io/secret/103c4c86-a9f6-4a59-a615-940717e39dc9-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716143    4428 manager.go:319] Starting recovery of all containers
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716149    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f97d160-4ad4-4546-9310-4a20d1309648" volumeName="kubernetes.io/configmap/2f97d160-4ad4-4546-9310-4a20d1309648-serviceca" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716164    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="575eac51-2cde-4d41-9de7-f6987fb83dfe" volumeName="kubernetes.io/projected/575eac51-2cde-4d41-9de7-f6987fb83dfe-kube-api-access-64rv7" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716181    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1b19e39-5b5d-46f7-b005-a1e5165fe135" volumeName="kubernetes.io/secret/c1b19e39-5b5d-46f7-b005-a1e5165fe135-agent-certs" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716196    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62aae01d-fd1d-43c3-9ad7-965bc6d595e2" volumeName="kubernetes.io/projected/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-kube-api-access-s5hcb" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716212    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="852969d3-8684-454c-ac42-56f80877702a" volumeName="kubernetes.io/configmap/852969d3-8684-454c-ac42-56f80877702a-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716224    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d617e36-a72a-4a8c-bc9c-85500f109e42" volumeName="kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716238    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb046f36-d918-4cdd-b2be-f2eec97ae74e" volumeName="kubernetes.io/empty-dir/cb046f36-d918-4cdd-b2be-f2eec97ae74e-tmp-dir" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716250    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb046f36-d918-4cdd-b2be-f2eec97ae74e" volumeName="kubernetes.io/projected/cb046f36-d918-4cdd-b2be-f2eec97ae74e-kube-api-access-9rnl5" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716262    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7a89a36-092a-4d38-8a50-60619cc50832" volumeName="kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716294    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f24ca8d4-b321-4579-aff7-e1d0177f50bc" volumeName="kubernetes.io/configmap/f24ca8d4-b321-4579-aff7-e1d0177f50bc-cni-binary-copy" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716311    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12e357d7-757d-4f89-837e-6283d4f87e11" volumeName="kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716329    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62aae01d-fd1d-43c3-9ad7-965bc6d595e2" volumeName="kubernetes.io/empty-dir/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-tmp-dir" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716342    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82689b74-1468-4087-8123-b361839b4124" volumeName="kubernetes.io/projected/82689b74-1468-4087-8123-b361839b4124-kube-api-access-tdl5n" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716354    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1b19e39-5b5d-46f7-b005-a1e5165fe135" volumeName="kubernetes.io/configmap/c1b19e39-5b5d-46f7-b005-a1e5165fe135-konnectivity-ca" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716365    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f24ca8d4-b321-4579-aff7-e1d0177f50bc" volumeName="kubernetes.io/projected/f24ca8d4-b321-4579-aff7-e1d0177f50bc-kube-api-access-m78mh" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716378    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="103c4c86-a9f6-4a59-a615-940717e39dc9" volumeName="kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-ovnkube-config" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716390    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f1fbe3-7baa-4d79-b758-5c5cdc99f45d" volumeName="kubernetes.io/configmap/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-iptables-alerter-script" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716402    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb046f36-d918-4cdd-b2be-f2eec97ae74e" volumeName="kubernetes.io/configmap/cb046f36-d918-4cdd-b2be-f2eec97ae74e-config-volume" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716415    4428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f24ca8d4-b321-4579-aff7-e1d0177f50bc" volumeName="kubernetes.io/configmap/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-daemon-config" seLinuxMountContext=""
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716428    4428 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 13:03:22.716431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716436    4428 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 13:03:22.717987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.716932    4428 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:03:22.718765 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.718745    4428 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:03:22.726056 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.726039    4428 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:03:22.729440 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.729418    4428 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 13:03:22.745546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.745489    4428 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 13:03:22.747139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.747112    4428 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 13:03:22.747139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.747136    4428 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 13:03:22.747284 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.747223    4428 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 13:03:22.747338 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:03:22.747275    4428 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 13:03:22.751687 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.751664    4428 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:03:22.769465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.769434    4428 generic.go:356] "Generic (PLEG): container finished" podID="8d617e36-a72a-4a8c-bc9c-85500f109e42" containerID="80b51cbabe9267284425d3b962dd83d284b59cc2e1800546c0be1c273232ac57" exitCode=0
Oct 31 13:03:22.769465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.769466    4428 generic.go:356] "Generic (PLEG): container finished" podID="8d617e36-a72a-4a8c-bc9c-85500f109e42" containerID="8239bb306218a8c34e9a8211de5baf8047690d25913007a9f3c1914487ac4407" exitCode=0
Oct 31 13:03:22.769605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.769477    4428 generic.go:356] "Generic (PLEG): container finished" podID="8d617e36-a72a-4a8c-bc9c-85500f109e42" containerID="f505b7ed1b44378b61fc85db85953c9cdab8e8d15ba1b9efb119cccdf08de270" exitCode=0
Oct 31 13:03:22.769605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.769486    4428 generic.go:356] "Generic (PLEG): container finished" podID="8d617e36-a72a-4a8c-bc9c-85500f109e42" containerID="f208a9e7998f2a5cb093ca67c4d755403298508c1d986c7aa1ad600f03b4e0ee" exitCode=0
Oct 31 13:03:22.769605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.769495    4428 generic.go:356] "Generic (PLEG): container finished" podID="8d617e36-a72a-4a8c-bc9c-85500f109e42" containerID="b3d75dd886e3341305bd56487c668509b50d9a29015dffc2d982ca147197fb87" exitCode=0
Oct 31 13:03:22.769605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.769504    4428 generic.go:356] "Generic (PLEG): container finished" podID="8d617e36-a72a-4a8c-bc9c-85500f109e42" containerID="36caeef49cd719e6dd70cd18ce0b0cee2f25808d95a629bdaf553a2f57b0a5d7" exitCode=0
Oct 31 13:03:22.782003 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.781956    4428 manager.go:324] Recovery completed
Oct 31 13:03:22.784579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.784549    4428 generic.go:356] "Generic (PLEG): container finished" podID="91e8205bd2503f1072bc57fd6d59d36b" containerID="d7b5039557c7f7f324f10ea016af84435d15f1b00d3de57d9b575d616fb26b65" exitCode=0
Oct 31 13:03:22.795816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.795796    4428 cpu_manager.go:226] "Starting" policy="none"
Oct 31 13:03:22.795816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.795808    4428 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 13:03:22.795977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.795824    4428 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 13:03:22.795977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.795968    4428 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 13:03:22.796078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.795979    4428 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 13:03:22.796078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.796048    4428 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 13:03:22.796078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.796055    4428 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 13:03:22.796078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.796060    4428 policy_none.go:50] "Start"
Oct 31 13:03:22.796078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.796069    4428 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 13:03:22.796078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.796077    4428 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 13:03:22.796240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.796198    4428 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 13:03:22.796240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.796205    4428 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 13:03:22.796240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.796210    4428 policy_none.go:44] "Start"
Oct 31 13:03:22.800400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.799923    4428 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 13:03:22.805389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.805375    4428 manager.go:341] "Starting Device Plugin manager"
Oct 31 13:03:22.805482 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:03:22.805400    4428 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 13:03:22.805482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.805409    4428 server.go:87] "Starting device plugin registration server"
Oct 31 13:03:22.805713 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.805700    4428 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 13:03:22.805747 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.805719    4428 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 13:03:22.805845 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.805824    4428 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 13:03:22.805964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.805927    4428 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 13:03:22.805964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.805956    4428 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 13:03:22.805964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.805960    4428 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 13:03:22.806129 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.806007    4428 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T13:03:22.805948031Z","UUID":"cfee5838-35db-41d7-ba5e-7f92f2d9d64a","Handler":null,"Name":"","Endpoint":""}
Oct 31 13:03:22.807084 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:03:22.807055    4428 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 13:03:22.807200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.807182    4428 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 13:03:22.807255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.807229    4428 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 13:03:22.847791 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.847756    4428 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal"]
Oct 31 13:03:22.848268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.848239    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal" event={"ID":"62681df534845faea6eef1fcf975d228","Type":"ContainerStarted","Data":"1197badf701b04f662e2cc986cf0daa70518d5d993136e0dddbb7e88947b40d3"}
Oct 31 13:03:22.848366 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.848281    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal" event={"ID":"62681df534845faea6eef1fcf975d228","Type":"ContainerStarted","Data":"d0007b66a1deb53d8b3459e5c7a8d692ea642205e62878bc20b1f25b1955d151"}
Oct 31 13:03:22.848366 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.848314    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal" event={"ID":"91e8205bd2503f1072bc57fd6d59d36b","Type":"ContainerStarted","Data":"6308a07606b9fc75bdc1f8a01adaff0c39fe66ac3f32d4c6bbd073b682d1dd56"}
Oct 31 13:03:22.848366 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.848318    4428 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:22.848366 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.848328    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal" event={"ID":"91e8205bd2503f1072bc57fd6d59d36b","Type":"ContainerDied","Data":"d7b5039557c7f7f324f10ea016af84435d15f1b00d3de57d9b575d616fb26b65"}
Oct 31 13:03:22.848366 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.848346    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal" event={"ID":"91e8205bd2503f1072bc57fd6d59d36b","Type":"ContainerStarted","Data":"c82bdb7624c5c7e1724272d83a313273acd800ceae755fe5179ccfab6d9d1ff8"}
Oct 31 13:03:22.848511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.848429    4428 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:22.861616 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.861596    4428 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 13:03:22.861669 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:03:22.861636    4428 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:22.862563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.862552    4428 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 13:03:22.862606 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:03:22.862582    4428 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:22.906694 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.906677    4428 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:03:22.907895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.907862    4428 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:03:22.907967 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.907907    4428 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:03:22.907967 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.907918    4428 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-168.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:03:22.907967 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.907943    4428 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:22.923224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.923205    4428 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:22.923299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.923252    4428 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:22.930496 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.930475    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62681df534845faea6eef1fcf975d228-config\") pod \"kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"62681df534845faea6eef1fcf975d228\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:22.930546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.930503    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91e8205bd2503f1072bc57fd6d59d36b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"91e8205bd2503f1072bc57fd6d59d36b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:22.930546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:22.930521    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91e8205bd2503f1072bc57fd6d59d36b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"91e8205bd2503f1072bc57fd6d59d36b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:23.031208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.031184    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62681df534845faea6eef1fcf975d228-config\") pod \"kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"62681df534845faea6eef1fcf975d228\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:23.031320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.031218    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91e8205bd2503f1072bc57fd6d59d36b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"91e8205bd2503f1072bc57fd6d59d36b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:23.031320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.031236    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91e8205bd2503f1072bc57fd6d59d36b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"91e8205bd2503f1072bc57fd6d59d36b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:23.031320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.031291    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91e8205bd2503f1072bc57fd6d59d36b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"91e8205bd2503f1072bc57fd6d59d36b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:23.031429 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.031311    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91e8205bd2503f1072bc57fd6d59d36b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"91e8205bd2503f1072bc57fd6d59d36b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:23.031429 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.031291    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62681df534845faea6eef1fcf975d228-config\") pod \"kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal\" (UID: \"62681df534845faea6eef1fcf975d228\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:23.695971 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.695935    4428 apiserver.go:52] "Watching apiserver"
Oct 31 13:03:23.704329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.704306    4428 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 13:03:23.704824 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.704804    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kstrj","openshift-image-registry/node-ca-nmwhw","openshift-insights/insights-runtime-extractor-f2kwh","openshift-dns/node-resolver-z65rw","openshift-cluster-node-tuning-operator/tuned-rzs7m","openshift-multus/network-metrics-daemon-s49pl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-swfl5","openshift-network-diagnostics/network-check-target-xpskx","kube-system/konnectivity-agent-ff7kw","openshift-network-operator/iptables-alerter-7bpj5","kube-system/global-pull-secret-syncer-6tcxq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9","openshift-multus/multus-additional-cni-plugins-x6wvd","kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal","openshift-multus/multus-dxmqp","openshift-ingress-canary/ingress-canary-jnzs8","openshift-ovn-kubernetes/ovnkube-node-vl2d9"]
Oct 31 13:03:23.709338 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.709314    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:03:23.710231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.710186    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 13:03:23.710231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.710188    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:03:23.710231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.710220    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 13:03:23.710231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.710192    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j8xvf\""
Oct 31 13:03:23.710493 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.710234    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 13:03:23.710493 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.710309    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkxw2\""
Oct 31 13:03:23.710493 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.710366    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 13:03:23.712007 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.711989    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 13:03:23.712130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.712099    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:23.712453 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.712430    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r2wwf\""
Oct 31 13:03:23.713247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.713227    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 13:03:23.713342 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.713258    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 13:03:23.713425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.713409    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:23.714942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.714928    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 13:03:23.714942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.714936    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:03:23.715110 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.715094    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 13:03:23.715165 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.715102    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 13:03:23.715210 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.715164    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 13:03:23.715315 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.715298    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 13:03:23.715373 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.715322    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 13:03:23.715500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.715459    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 13:03:23.715590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.715556    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s8mtw\""
Oct 31 13:03:23.715676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.715649    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 13:03:23.715710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.715686    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 13:03:23.715710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.715704    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 13:03:23.715789 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.715724    4428 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 13:03:23.716305 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716293    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 13:03:23.716358 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716347    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 13:03:23.716418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716374    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 13:03:23.716418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716378    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-65v8h\""
Oct 31 13:03:23.716418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716398    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 13:03:23.716654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716635    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 13:03:23.716712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716678    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 13:03:23.716712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716699    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tmkp5\""
Oct 31 13:03:23.716838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716701    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 13:03:23.716838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716760    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 13:03:23.716838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716778    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lxhpt\""
Oct 31 13:03:23.716838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716782    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 13:03:23.716838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716779    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wr5k7\""
Oct 31 13:03:23.716838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716779    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-s9d6g\""
Oct 31 13:03:23.716838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716802    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 13:03:23.716838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716812    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-h72qs\""
Oct 31 13:03:23.716838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716803    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 13:03:23.716838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716816    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5rqjw\""
Oct 31 13:03:23.717152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.716886    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-glfvs\""
Oct 31 13:03:23.718615 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.718600    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 13:03:23.718845 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.718815    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 13:03:23.718845 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.718841    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 13:03:23.718980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.718841    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 13:03:23.718980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.718862    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 13:03:23.718980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.718915    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 13:03:23.718980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.718922    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 13:03:23.718980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.718922    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 13:03:23.719117 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.719000    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 13:03:23.735921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.735897    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-device-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.735994 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.735933    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-systemd\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.735994 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.735956    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-var-lib-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.736084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.736051    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7cnz\" (UniqueName: \"kubernetes.io/projected/103c4c86-a9f6-4a59-a615-940717e39dc9-kube-api-access-f7cnz\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.736120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.736103    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-netns\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.736166 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.736134    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:23.736216 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.736201    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-lib-modules\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.736262 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.736242    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-kubelet\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.736300 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.736275    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-systemd-units\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.736334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.736300    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-host-slash\") pod \"iptables-alerter-7bpj5\" (UID: \"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d\") " pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:03:23.736334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.736326    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-cni-bin\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.736403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.736345    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rnl5\" (UniqueName: \"kubernetes.io/projected/cb046f36-d918-4cdd-b2be-f2eec97ae74e-kube-api-access-9rnl5\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:23.736403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.736390    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-run\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.736482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.736416    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f24ca8d4-b321-4579-aff7-e1d0177f50bc-cni-binary-copy\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.736567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.736551    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-daemon-config\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.736701 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.736667    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rv7\" (UniqueName: \"kubernetes.io/projected/575eac51-2cde-4d41-9de7-f6987fb83dfe-kube-api-access-64rv7\") pod \"ingress-canary-jnzs8\" (UID: \"575eac51-2cde-4d41-9de7-f6987fb83dfe\") " pod="openshift-ingress-canary/ingress-canary-jnzs8"
Oct 31 13:03:23.736821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.736792    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/852969d3-8684-454c-ac42-56f80877702a-crio-socket\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:23.736988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.736972    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/852969d3-8684-454c-ac42-56f80877702a-data-volume\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:23.737056 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.736996    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852969d3-8684-454c-ac42-56f80877702a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:23.737056 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.737020    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-daemon-config\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.737154 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.737060    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-cni-bin\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.737154 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.737053    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f24ca8d4-b321-4579-aff7-e1d0177f50bc-cni-binary-copy\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.737154 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.737099    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/852969d3-8684-454c-ac42-56f80877702a-data-volume\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:23.737154 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.737127    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-env-overrides\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.737309 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.737179    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb046f36-d918-4cdd-b2be-f2eec97ae74e-config-volume\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:23.737309 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.737209    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7a89a36-092a-4d38-8a50-60619cc50832-kubelet-config\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:23.737309 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.737253    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-kubernetes\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.737827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.737295    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-ovnkube-script-lib\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.737910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.737851    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-system-cni-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.737910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.737888    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-env-overrides\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.738147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.737918    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-cni-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.738147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.737958    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852969d3-8684-454c-ac42-56f80877702a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:23.738147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.737985    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-cni-multus\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.738147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.738014    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-hostroot\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.738147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.738046    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-conf-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.738383 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.738194    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f97d160-4ad4-4546-9310-4a20d1309648-host\") pod \"node-ca-nmwhw\" (UID: \"2f97d160-4ad4-4546-9310-4a20d1309648\") " pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:03:23.738383 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.738273    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t24pv\" (UniqueName: \"kubernetes.io/projected/852969d3-8684-454c-ac42-56f80877702a-kube-api-access-t24pv\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:23.738383 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.738316    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-tmp-dir\") pod \"node-resolver-z65rw\" (UID: \"62aae01d-fd1d-43c3-9ad7-965bc6d595e2\") " pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:03:23.738383 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.738304    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb046f36-d918-4cdd-b2be-f2eec97ae74e-config-volume\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:23.738383 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.738366    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-systemd\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.738383 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.738372    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-ovnkube-script-lib\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.738651 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.738400    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-etc-selinux\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.738651 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.738440    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-tmp-dir\") pod \"node-resolver-z65rw\" (UID: \"62aae01d-fd1d-43c3-9ad7-965bc6d595e2\") " pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:03:23.738651 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.738508    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8s79\" (UniqueName: \"kubernetes.io/projected/d8d147e3-324f-416b-8155-4d3631c5667e-kube-api-access-s8s79\") pod \"migrator-66df5d496b-swfl5\" (UID: \"d8d147e3-324f-416b-8155-4d3631c5667e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-swfl5"
Oct 31 13:03:23.738651 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.738545    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-slash\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.738651 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.738636    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-run-ovn-kubernetes\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.738937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.738764    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-cni-netd\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.738937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.738815    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:03:23.738937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.738898    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-iptables-alerter-script\") pod \"iptables-alerter-7bpj5\" (UID: \"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d\") " pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:03:23.739078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.738935    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-k8s-cni-cncf-io\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.739078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.739001    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-tmp\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.739181 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.739124    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-tmp\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.739238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.739221    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs\") pod \"network-metrics-daemon-s49pl\" (UID: \"0edfba66-6a3a-4be8-b420-227d0443ddea\") " pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:03:23.739329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.739290    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhrzc\" (UniqueName: \"kubernetes.io/projected/0edfba66-6a3a-4be8-b420-227d0443ddea-kube-api-access-zhrzc\") pod \"network-metrics-daemon-s49pl\" (UID: \"0edfba66-6a3a-4be8-b420-227d0443ddea\") " pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:03:23.739382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.739339    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-iptables-alerter-script\") pod \"iptables-alerter-7bpj5\" (UID: \"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d\") " pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:03:23.739382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.739349    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1b19e39-5b5d-46f7-b005-a1e5165fe135-agent-certs\") pod \"konnectivity-agent-ff7kw\" (UID: \"c1b19e39-5b5d-46f7-b005-a1e5165fe135\") " pod="kube-system/konnectivity-agent-ff7kw"
Oct 31 13:03:23.739479 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.739429    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:03:23.739479 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.739469    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82sb9\" (UniqueName: \"kubernetes.io/projected/2f97d160-4ad4-4546-9310-4a20d1309648-kube-api-access-82sb9\") pod \"node-ca-nmwhw\" (UID: \"2f97d160-4ad4-4546-9310-4a20d1309648\") " pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:03:23.739577 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.739541    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-node-log\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.739632 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.739578    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.739801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.739786    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-cni-binary-copy\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:03:23.739909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.739859    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-host\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.739990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.739932    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xl4l\" (UniqueName: \"kubernetes.io/projected/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-kube-api-access-5xl4l\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.739990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.739963    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-registration-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.740094 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.739998    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/103c4c86-a9f6-4a59-a615-940717e39dc9-ovn-node-metrics-cert\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.740094 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.740080    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-system-cni-dir\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:03:23.740201 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.740140    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94j67\" (UniqueName: \"kubernetes.io/projected/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-kube-api-access-94j67\") pod \"iptables-alerter-7bpj5\" (UID: \"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d\") " pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:03:23.740201 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.740177    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-multus-certs\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.740299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.740213    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-socket-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.740370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.740333    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-sys-fs\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.740450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.740428    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmbrx\" (UniqueName: \"kubernetes.io/projected/8d617e36-a72a-4a8c-bc9c-85500f109e42-kube-api-access-bmbrx\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:03:23.740510 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.740459    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-kubelet\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.740510 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.740496    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb2z7\" (UniqueName: \"kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7\") pod \"network-check-target-xpskx\" (UID: \"12e357d7-757d-4f89-837e-6283d4f87e11\") " pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:03:23.740785 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.740743    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysctl-d\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.740882 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.740796    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1b19e39-5b5d-46f7-b005-a1e5165fe135-konnectivity-ca\") pod \"konnectivity-agent-ff7kw\" (UID: \"c1b19e39-5b5d-46f7-b005-a1e5165fe135\") " pod="kube-system/konnectivity-agent-ff7kw"
Oct 31 13:03:23.740882 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.740826    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-modprobe-d\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.740985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.740858    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysctl-conf\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.740985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.740926    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-run-netns\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.740985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.740975    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.741140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.741008    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-cnibin\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:03:23.741140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.741091    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:03:23.741140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.741128    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-tuned\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.741288 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.741230    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-etc-kubernetes\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.741340 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.741306    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb046f36-d918-4cdd-b2be-f2eec97ae74e-tmp-dir\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:23.741446 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.741358    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5hcb\" (UniqueName: \"kubernetes.io/projected/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-kube-api-access-s5hcb\") pod \"node-resolver-z65rw\" (UID: \"62aae01d-fd1d-43c3-9ad7-965bc6d595e2\") " pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:03:23.741446 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.741411    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysconfig\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.741590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.741576    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-var-lib-kubelet\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.741706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.741680    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-ovnkube-config\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.741779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.741732    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78mh\" (UniqueName: \"kubernetes.io/projected/f24ca8d4-b321-4579-aff7-e1d0177f50bc-kube-api-access-m78mh\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.741779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.741771    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-hosts-file\") pod \"node-resolver-z65rw\" (UID: \"62aae01d-fd1d-43c3-9ad7-965bc6d595e2\") " pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:03:23.741939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.741807    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-etc-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.742121 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.742077    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb046f36-d918-4cdd-b2be-f2eec97ae74e-tmp-dir\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:23.742218 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.742117    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-tuned\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.742265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.742195    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-ovn\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.742312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.742269    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1b19e39-5b5d-46f7-b005-a1e5165fe135-konnectivity-ca\") pod \"konnectivity-agent-ff7kw\" (UID: \"c1b19e39-5b5d-46f7-b005-a1e5165fe135\") " pod="kube-system/konnectivity-agent-ff7kw"
Oct 31 13:03:23.742312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.742291    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-log-socket\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.742421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.742400    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-socket-dir-parent\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.742523 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.742463    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1b19e39-5b5d-46f7-b005-a1e5165fe135-agent-certs\") pod \"konnectivity-agent-ff7kw\" (UID: \"c1b19e39-5b5d-46f7-b005-a1e5165fe135\") " pod="kube-system/konnectivity-agent-ff7kw"
Oct 31 13:03:23.742611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.742552    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f97d160-4ad4-4546-9310-4a20d1309648-serviceca\") pod \"node-ca-nmwhw\" (UID: \"2f97d160-4ad4-4546-9310-4a20d1309648\") " pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:03:23.742676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.742605    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/103c4c86-a9f6-4a59-a615-940717e39dc9-ovnkube-config\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.742720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.742692    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:23.743214 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.743192    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/575eac51-2cde-4d41-9de7-f6987fb83dfe-cert\") pod \"ingress-canary-jnzs8\" (UID: \"575eac51-2cde-4d41-9de7-f6987fb83dfe\") " pod="openshift-ingress-canary/ingress-canary-jnzs8"
Oct 31 13:03:23.743293 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.743269    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:03:23.743352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.743287    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7a89a36-092a-4d38-8a50-60619cc50832-dbus\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:23.743352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.743293    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f97d160-4ad4-4546-9310-4a20d1309648-serviceca\") pod \"node-ca-nmwhw\" (UID: \"2f97d160-4ad4-4546-9310-4a20d1309648\") " pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:03:23.743352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.743281    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:03:23.743352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.743328    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-sys\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.743352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.743340    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d617e36-a72a-4a8c-bc9c-85500f109e42-cni-binary-copy\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:03:23.743557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.743380    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.743557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.743425    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdl5n\" (UniqueName: \"kubernetes.io/projected/82689b74-1468-4087-8123-b361839b4124-kube-api-access-tdl5n\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.743557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.743475    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-cnibin\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.743699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.743559    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:23.743885 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.743852    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7a89a36-092a-4d38-8a50-60619cc50832-original-pull-secret\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:23.744004 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.743988    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/103c4c86-a9f6-4a59-a615-940717e39dc9-ovn-node-metrics-cert\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.747081 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.747060    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rv7\" (UniqueName: \"kubernetes.io/projected/575eac51-2cde-4d41-9de7-f6987fb83dfe-kube-api-access-64rv7\") pod \"ingress-canary-jnzs8\" (UID: \"575eac51-2cde-4d41-9de7-f6987fb83dfe\") " pod="openshift-ingress-canary/ingress-canary-jnzs8"
Oct 31 13:03:23.747327 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.747308    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rnl5\" (UniqueName: \"kubernetes.io/projected/cb046f36-d918-4cdd-b2be-f2eec97ae74e-kube-api-access-9rnl5\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:23.747365 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.747318    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7cnz\" (UniqueName: \"kubernetes.io/projected/103c4c86-a9f6-4a59-a615-940717e39dc9-kube-api-access-f7cnz\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.748563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.748542    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t24pv\" (UniqueName: \"kubernetes.io/projected/852969d3-8684-454c-ac42-56f80877702a-kube-api-access-t24pv\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:23.748816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.748801    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8s79\" (UniqueName: \"kubernetes.io/projected/d8d147e3-324f-416b-8155-4d3631c5667e-kube-api-access-s8s79\") pod \"migrator-66df5d496b-swfl5\" (UID: \"d8d147e3-324f-416b-8155-4d3631c5667e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-swfl5"
Oct 31 13:03:23.751993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.751975    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94j67\" (UniqueName: \"kubernetes.io/projected/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-kube-api-access-94j67\") pod \"iptables-alerter-7bpj5\" (UID: \"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d\") " pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:03:23.752183 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.752164    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhrzc\" (UniqueName: \"kubernetes.io/projected/0edfba66-6a3a-4be8-b420-227d0443ddea-kube-api-access-zhrzc\") pod \"network-metrics-daemon-s49pl\" (UID: \"0edfba66-6a3a-4be8-b420-227d0443ddea\") " pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:03:23.752281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.752263    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82sb9\" (UniqueName: \"kubernetes.io/projected/2f97d160-4ad4-4546-9310-4a20d1309648-kube-api-access-82sb9\") pod \"node-ca-nmwhw\" (UID: \"2f97d160-4ad4-4546-9310-4a20d1309648\") " pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:03:23.752439 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.752425    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmbrx\" (UniqueName: \"kubernetes.io/projected/8d617e36-a72a-4a8c-bc9c-85500f109e42-kube-api-access-bmbrx\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:03:23.752981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.752963    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb2z7\" (UniqueName: \"kubernetes.io/projected/12e357d7-757d-4f89-837e-6283d4f87e11-kube-api-access-kb2z7\") pod \"network-check-target-xpskx\" (UID: \"12e357d7-757d-4f89-837e-6283d4f87e11\") " pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:03:23.753022 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.752992    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xl4l\" (UniqueName: \"kubernetes.io/projected/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-kube-api-access-5xl4l\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.753174 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.753160    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78mh\" (UniqueName: \"kubernetes.io/projected/f24ca8d4-b321-4579-aff7-e1d0177f50bc-kube-api-access-m78mh\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.753365 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.753345    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5hcb\" (UniqueName: \"kubernetes.io/projected/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-kube-api-access-s5hcb\") pod \"node-resolver-z65rw\" (UID: \"62aae01d-fd1d-43c3-9ad7-965bc6d595e2\") " pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:03:23.753932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.753917    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdl5n\" (UniqueName: \"kubernetes.io/projected/82689b74-1468-4087-8123-b361839b4124-kube-api-access-tdl5n\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.788266 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.788243    4428 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:23.788425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.788411    4428 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:23.801156 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.801134    4428 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 13:03:23.801222 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:03:23.801175    4428 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:23.801767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.801752    4428 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 13:03:23.801840 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:03:23.801794    4428 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-168.us-east-2.compute.internal"
Oct 31 13:03:23.810062 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.810046    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 13:03:23.810163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.810150    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 13:03:23.810376 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.810361    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 13:03:23.810432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.810376    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-66zbj\""
Oct 31 13:03:23.810432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.810405    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tmfz7\""
Oct 31 13:03:23.810535 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.810453    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qbkfg\""
Oct 31 13:03:23.810535 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.810507    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wdnw7\""
Oct 31 13:03:23.813693 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.813680    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/575eac51-2cde-4d41-9de7-f6987fb83dfe-cert\") pod \"ingress-canary-jnzs8\" (UID: \"575eac51-2cde-4d41-9de7-f6987fb83dfe\") " pod="openshift-ingress-canary/ingress-canary-jnzs8"
Oct 31 13:03:23.835281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.835246    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:23.844642 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844609    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-systemd\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.844642 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844644    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-etc-selinux\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.844902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844665    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-slash\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.844902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844681    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-run-ovn-kubernetes\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.844902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844734    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-slash\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.844902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844745    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-systemd\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.844902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844748    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-etc-selinux\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.844902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844772    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-run-ovn-kubernetes\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.844902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844795    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-cni-netd\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.844902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844820    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:03:23.844902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844837    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-k8s-cni-cncf-io\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.844902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844852    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs\") pod \"network-metrics-daemon-s49pl\" (UID: \"0edfba66-6a3a-4be8-b420-227d0443ddea\") " pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:03:23.844902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844843    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-cni-netd\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.844902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844894    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-node-log\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.844902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844912    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.844902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844912    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-k8s-cni-cncf-io\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.844902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844909    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:03:23.844902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.844932    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-host\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845003    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845016    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-node-log\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845016    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-host\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845060    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-registration-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845097    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-system-cni-dir\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845122    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-multus-certs\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845130    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-registration-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845153    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-socket-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845172    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-system-cni-dir\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845172    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-sys-fs\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845214    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-sys-fs\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845229    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-socket-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845216    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-multus-certs\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845247    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-kubelet\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845257    4428 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845283    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysctl-d\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845312    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-modprobe-d\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845326    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-kubelet\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845337    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysctl-conf\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845368    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-run-netns\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845396    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845414    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-modprobe-d\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845419    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-cnibin\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845425    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysctl-d\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845445    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysctl-conf\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845451    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-etc-kubernetes\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845460    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d617e36-a72a-4a8c-bc9c-85500f109e42-cnibin\") pod \"multus-additional-cni-plugins-x6wvd\" (UID: \"8d617e36-a72a-4a8c-bc9c-85500f109e42\") " pod="openshift-multus/multus-additional-cni-plugins-x6wvd"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845472    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-run-netns\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845476    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845490    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysconfig\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845508    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-etc-kubernetes\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845524    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-sysconfig\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.845517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845530    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-var-lib-kubelet\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845570    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-var-lib-kubelet\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845610    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-hosts-file\") pod \"node-resolver-z65rw\" (UID: \"62aae01d-fd1d-43c3-9ad7-965bc6d595e2\") " pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845639    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-etc-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845664    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-ovn\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845680    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/62aae01d-fd1d-43c3-9ad7-965bc6d595e2-hosts-file\") pod \"node-resolver-z65rw\" (UID: \"62aae01d-fd1d-43c3-9ad7-965bc6d595e2\") " pod="openshift-dns/node-resolver-z65rw"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845683    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-etc-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845687    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-log-socket\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845706    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-ovn\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845712    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-socket-dir-parent\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845723    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-log-socket\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845730    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845777    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-socket-dir-parent\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:03:23.845802    4428 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845803    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7a89a36-092a-4d38-8a50-60619cc50832-dbus\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845859    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7a89a36-092a-4d38-8a50-60619cc50832-dbus\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:03:23.845903    4428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls podName:cb046f36-d918-4cdd-b2be-f2eec97ae74e nodeName:}" failed. No retries permitted until 2026-10-31 13:03:24.345863285 +0000 UTC m=+1.744211923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls") pod "dns-default-kstrj" (UID: "cb046f36-d918-4cdd-b2be-f2eec97ae74e") : secret "dns-default-metrics-tls" not found
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845917    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-sys\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845935    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845954    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-cnibin\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845971    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-device-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.845993    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-systemd\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846011    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846021    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-var-lib-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846005    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-sys\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846038    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-cnibin\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846048    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-netns\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846054    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-run-systemd\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846075    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-var-lib-openvswitch\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846062    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82689b74-1468-4087-8123-b361839b4124-device-dir\") pod \"aws-ebs-csi-driver-node-ww8c9\" (UID: \"82689b74-1468-4087-8123-b361839b4124\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ww8c9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846082    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-run-netns\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846084    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846125    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-lib-modules\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:03:23.846141    4428 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846161    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-kubelet\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846186    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-systemd-units\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:03:23.846190    4428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls podName:852969d3-8684-454c-ac42-56f80877702a nodeName:}" failed. No retries permitted until 2026-10-31 13:03:24.346179391 +0000 UTC m=+1.744527993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2kwh" (UID: "852969d3-8684-454c-ac42-56f80877702a") : secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846218    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-host-slash\") pod \"iptables-alerter-7bpj5\" (UID: \"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d\") " pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846220    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-lib-modules\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846232    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-cni-bin\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846227    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-kubelet\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846263    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-run\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846275    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75f1fbe3-7baa-4d79-b758-5c5cdc99f45d-host-slash\") pod \"iptables-alerter-7bpj5\" (UID: \"75f1fbe3-7baa-4d79-b758-5c5cdc99f45d\") " pod="openshift-network-operator/iptables-alerter-7bpj5"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846226    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-systemd-units\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846298    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/852969d3-8684-454c-ac42-56f80877702a-crio-socket\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846307    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-run\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846320    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-cni-bin\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846333    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-cni-bin\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846349    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/852969d3-8684-454c-ac42-56f80877702a-crio-socket\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846359    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7a89a36-092a-4d38-8a50-60619cc50832-kubelet-config\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846365    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/103c4c86-a9f6-4a59-a615-940717e39dc9-host-cni-bin\") pod \"ovnkube-node-vl2d9\" (UID: \"103c4c86-a9f6-4a59-a615-940717e39dc9\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846396    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-kubernetes\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846405    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7a89a36-092a-4d38-8a50-60619cc50832-kubelet-config\") pod \"global-pull-secret-syncer-6tcxq\" (UID: \"d7a89a36-092a-4d38-8a50-60619cc50832\") " pod="kube-system/global-pull-secret-syncer-6tcxq"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846421    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-system-cni-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846436    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46-etc-kubernetes\") pod \"tuned-rzs7m\" (UID: \"5e0b8381-9ec1-4e03-a025-d3f1d7ff7c46\") " pod="openshift-cluster-node-tuning-operator/tuned-rzs7m"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846462    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-cni-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846478    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-system-cni-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846483    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-cni-multus\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846499    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-hostroot\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846513    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-conf-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846525    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-host-var-lib-cni-multus\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846534    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f97d160-4ad4-4546-9310-4a20d1309648-host\") pod \"node-ca-nmwhw\" (UID: \"2f97d160-4ad4-4546-9310-4a20d1309648\") " pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846531    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-cni-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846564    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-hostroot\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846567    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f24ca8d4-b321-4579-aff7-e1d0177f50bc-multus-conf-dir\") pod \"multus-dxmqp\" (UID: \"f24ca8d4-b321-4579-aff7-e1d0177f50bc\") " pod="openshift-multus/multus-dxmqp"
Oct 31 13:03:23.846803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.846598    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f97d160-4ad4-4546-9310-4a20d1309648-host\") pod \"node-ca-nmwhw\" (UID: \"2f97d160-4ad4-4546-9310-4a20d1309648\") " pod="openshift-image-registry/node-ca-nmwhw"
Oct 31 13:03:23.848576 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:23.847586    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0edfba66-6a3a-4be8-b420-227d0443ddea-metrics-certs\") pod \"network-metrics-daemon-s49pl\" (UID: \"0edfba66-6a3a-4be8-b420-227d0443ddea\") " pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:03:24.007501 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:24.007485    4428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:03:24.009749 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:24.009732    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s49pl"
Oct 31 13:03:24.234858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:24.234823    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:24.314139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:24.314075    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s49pl"]
Oct 31 13:03:24.317156 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:24.317121    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0edfba66_6a3a_4be8_b420_227d0443ddea.slice/crio-81e88b957da5a1d4981a6c9e0caf6a8c1d5bed048ac16ef72236c630d64c2dbb WatchSource:0}: Error finding container 81e88b957da5a1d4981a6c9e0caf6a8c1d5bed048ac16ef72236c630d64c2dbb: Status 404 returned error can't find the container with id 81e88b957da5a1d4981a6c9e0caf6a8c1d5bed048ac16ef72236c630d64c2dbb
Oct 31 13:03:24.349747 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:24.349726    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:24.349845 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:24.349806    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:24.349927 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:03:24.349887    4428 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:24.349927 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:03:24.349920    4428 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 13:03:24.350017 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:03:24.349938    4428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls podName:852969d3-8684-454c-ac42-56f80877702a nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.349925012 +0000 UTC m=+2.748273626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2kwh" (UID: "852969d3-8684-454c-ac42-56f80877702a") : secret "insights-runtime-extractor-tls" not found
Oct 31 13:03:24.350017 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:03:24.349977    4428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls podName:cb046f36-d918-4cdd-b2be-f2eec97ae74e nodeName:}" failed. No retries permitted until 2026-10-31 13:03:25.349955328 +0000 UTC m=+2.748303950 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls") pod "dns-default-kstrj" (UID: "cb046f36-d918-4cdd-b2be-f2eec97ae74e") : secret "dns-default-metrics-tls" not found
Oct 31 13:03:24.429516 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:24.429495    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:24.615539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:24.615493    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z65rw_62aae01d-fd1d-43c3-9ad7-965bc6d595e2/dns-node-resolver/0.log"
Oct 31 13:03:24.791536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:24.791507    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xpskx" event={"ID":"12e357d7-757d-4f89-837e-6283d4f87e11","Type":"ContainerStarted","Data":"7c0843942b3784411d6c131357b2bb04f1f677ad80a0098cd61b6970328296fc"}
Oct 31 13:03:24.792633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:24.792607    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s49pl" event={"ID":"0edfba66-6a3a-4be8-b420-227d0443ddea","Type":"ContainerStarted","Data":"81e88b957da5a1d4981a6c9e0caf6a8c1d5bed048ac16ef72236c630d64c2dbb"}
Oct 31 13:03:25.030791 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:25.030756    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:25.356151 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:25.356054    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:25.356151 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:25.356115    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:25.359202 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:25.359179    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb046f36-d918-4cdd-b2be-f2eec97ae74e-metrics-tls\") pod \"dns-default-kstrj\" (UID: \"cb046f36-d918-4cdd-b2be-f2eec97ae74e\") " pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:25.359300 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:25.359219    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852969d3-8684-454c-ac42-56f80877702a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2kwh\" (UID: \"852969d3-8684-454c-ac42-56f80877702a\") " pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:25.512848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:25.512815    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:25.513918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:25.513899    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f2kwh"
Oct 31 13:03:25.614303 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:25.614249    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:25.618373 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:25.618353    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nmwhw_2f97d160-4ad4-4546-9310-4a20d1309648/node-ca/0.log"
Oct 31 13:03:25.761100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:25.761072    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:25.795933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:25.795913    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kstrj"]
Oct 31 13:03:25.799970 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:25.799932    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb046f36_d918_4cdd_b2be_f2eec97ae74e.slice/crio-f989238a6d8267961d364525e8de518e1e4b18198f6245d1d5a85ca50c0555c0 WatchSource:0}: Error finding container f989238a6d8267961d364525e8de518e1e4b18198f6245d1d5a85ca50c0555c0: Status 404 returned error can't find the container with id f989238a6d8267961d364525e8de518e1e4b18198f6245d1d5a85ca50c0555c0
Oct 31 13:03:25.817926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:25.817901    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f2kwh"]
Oct 31 13:03:25.822916 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:25.822890    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852969d3_8684_454c_ac42_56f80877702a.slice/crio-c633b95edcdc53275923348c5c024bb9b08aa1e00c07731409bb36919cb37cb7 WatchSource:0}: Error finding container c633b95edcdc53275923348c5c024bb9b08aa1e00c07731409bb36919cb37cb7: Status 404 returned error can't find the container with id c633b95edcdc53275923348c5c024bb9b08aa1e00c07731409bb36919cb37cb7
Oct 31 13:03:25.921768 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:25.921721    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:26.215418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:26.215380    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jnzs8_575eac51-2cde-4d41-9de7-f6987fb83dfe/serve-healthcheck-canary/0.log"
Oct 31 13:03:26.237498 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:26.237473    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:26.428341 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:26.428304    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:26.531095 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:26.531033    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:03:26.801161 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:26.801072    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s49pl" event={"ID":"0edfba66-6a3a-4be8-b420-227d0443ddea","Type":"ContainerStarted","Data":"ccf02cf49fce8adeba66d09615d0aa1224146dfa620f2a93041b5be1c4f4bd54"}
Oct 31 13:03:26.801161 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:26.801138    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s49pl" event={"ID":"0edfba66-6a3a-4be8-b420-227d0443ddea","Type":"ContainerStarted","Data":"b17681e66973095da6435edd9b8a4aefdb160ad5ae2ffe8828eb554f59f6bf2a"}
Oct 31 13:03:26.803396 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:26.803362    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2kwh" event={"ID":"852969d3-8684-454c-ac42-56f80877702a","Type":"ContainerStarted","Data":"707175e79ab4123644dd795845b7971f5235bf860080ce1a1d9de1cfde964827"}
Oct 31 13:03:26.803569 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:26.803422    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2kwh" event={"ID":"852969d3-8684-454c-ac42-56f80877702a","Type":"ContainerStarted","Data":"af561ef73cef9193a4e2aeac19c9d3e97258770f9ee46d05f5f2b7b946287df9"}
Oct 31 13:03:26.803569 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:26.803441    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2kwh" event={"ID":"852969d3-8684-454c-ac42-56f80877702a","Type":"ContainerStarted","Data":"c633b95edcdc53275923348c5c024bb9b08aa1e00c07731409bb36919cb37cb7"}
Oct 31 13:03:26.804795 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:26.804763    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kstrj" event={"ID":"cb046f36-d918-4cdd-b2be-f2eec97ae74e","Type":"ContainerStarted","Data":"f989238a6d8267961d364525e8de518e1e4b18198f6245d1d5a85ca50c0555c0"}
Oct 31 13:03:26.815847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:26.815828    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-swfl5_d8d147e3-324f-416b-8155-4d3631c5667e/migrator/0.log"
Oct 31 13:03:27.031901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:27.031783    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:27.391189 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:27.391155    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:27.630349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:27.630319    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:27.813139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:27.813109    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kstrj" event={"ID":"cb046f36-d918-4cdd-b2be-f2eec97ae74e","Type":"ContainerStarted","Data":"9f6e39dfe198ce0f5839c7f58bc2c3bcee545a6506d469f29fe7630fdc232eb6"}
Oct 31 13:03:27.813455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:27.813199    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kstrj" event={"ID":"cb046f36-d918-4cdd-b2be-f2eec97ae74e","Type":"ContainerStarted","Data":"5e499992d6a1d2eeb07174faa0937819c0333cd9d176af97608137df1d2e13d9"}
Oct 31 13:03:27.813455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:27.813248    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:27.829913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:27.829891    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:28.235821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:28.235796    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:28.433981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:28.433950    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:28.594117 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:28.594093    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:28.818710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:28.818673    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2kwh" event={"ID":"852969d3-8684-454c-ac42-56f80877702a","Type":"ContainerStarted","Data":"33a66305fac8b0011aca3bcde623ac930096e8538761638dc3003fa163b9017f"}
Oct 31 13:03:29.032612 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:29.032578    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:29.180987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:29.180947    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:29.436572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:29.436508    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:29.632281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:29.632252    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:29.786589 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:29.786565    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:29.786783 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:29.786770    4428 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 13:03:29.829649 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:29.829613    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:30.228601 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:30.228571    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:30.431557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:30.431526    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:31.031910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:31.031862    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:31.640508 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:31.640480    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:31.800375 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:31.800340    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:31.956822 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:31.956759    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:32.320399 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:32.320377    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:32.495514 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:32.495484    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:32.647888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:32.647799    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:32.818579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:32.818557    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:32.982207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:32.982183    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:33.178051 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:33.178028    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:33.331082 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:33.331011    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:34.083546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:34.083517    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:34.236561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:34.236526    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:34.394822 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:34.394746    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:34.500447 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:34.500421    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ff7kw"
Oct 31 13:03:34.545842 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:34.545819    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:35.175610 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:35.175580    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:35.330930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:35.330900    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:37.312238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:37.312205    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:37.468612 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:37.468582    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:38.824607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:38.824583    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kstrj"
Oct 31 13:03:40.485809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.485777    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68f55d596d-9ppzt"]
Oct 31 13:03:40.488739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.488721    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.500241 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.500219    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 13:03:40.520229 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.520203    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68f55d596d-9ppzt"]
Oct 31 13:03:40.552574 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.552551    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22cf051d-2e64-45a4-bb98-074e25a7769c-installation-pull-secrets\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.552675 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.552581    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qls4c\" (UniqueName: \"kubernetes.io/projected/22cf051d-2e64-45a4-bb98-074e25a7769c-kube-api-access-qls4c\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.552675 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.552600    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22cf051d-2e64-45a4-bb98-074e25a7769c-ca-trust-extracted\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.552744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.552672    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22cf051d-2e64-45a4-bb98-074e25a7769c-registry-tls\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.552744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.552694    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22cf051d-2e64-45a4-bb98-074e25a7769c-registry-certificates\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.552744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.552720    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22cf051d-2e64-45a4-bb98-074e25a7769c-bound-sa-token\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.552918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.552844    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22cf051d-2e64-45a4-bb98-074e25a7769c-image-registry-private-configuration\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.552918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.552905    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22cf051d-2e64-45a4-bb98-074e25a7769c-trusted-ca\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.588510 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.588489    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 13:03:40.588606 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.588512    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 13:03:40.588660 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.588643    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 13:03:40.588694 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.588650    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rw847\""
Oct 31 13:03:40.644902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.644859    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:40.653484 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.653466    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22cf051d-2e64-45a4-bb98-074e25a7769c-image-registry-private-configuration\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.653561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.653495    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22cf051d-2e64-45a4-bb98-074e25a7769c-trusted-ca\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.653561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.653515    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22cf051d-2e64-45a4-bb98-074e25a7769c-installation-pull-secrets\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.653561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.653530    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qls4c\" (UniqueName: \"kubernetes.io/projected/22cf051d-2e64-45a4-bb98-074e25a7769c-kube-api-access-qls4c\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.653561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.653549    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22cf051d-2e64-45a4-bb98-074e25a7769c-ca-trust-extracted\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.653751 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.653572    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22cf051d-2e64-45a4-bb98-074e25a7769c-registry-tls\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.653806 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.653774    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22cf051d-2e64-45a4-bb98-074e25a7769c-registry-certificates\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.653860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.653839    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22cf051d-2e64-45a4-bb98-074e25a7769c-bound-sa-token\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.654029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.654007    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22cf051d-2e64-45a4-bb98-074e25a7769c-ca-trust-extracted\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.654795 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.654775    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22cf051d-2e64-45a4-bb98-074e25a7769c-registry-certificates\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.654984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.654962    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22cf051d-2e64-45a4-bb98-074e25a7769c-trusted-ca\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.656077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.656057    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22cf051d-2e64-45a4-bb98-074e25a7769c-image-registry-private-configuration\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.656245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.656228    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22cf051d-2e64-45a4-bb98-074e25a7769c-installation-pull-secrets\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.656303 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.656241    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22cf051d-2e64-45a4-bb98-074e25a7769c-registry-tls\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.666838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.666814    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qls4c\" (UniqueName: \"kubernetes.io/projected/22cf051d-2e64-45a4-bb98-074e25a7769c-kube-api-access-qls4c\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.667106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.667090    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22cf051d-2e64-45a4-bb98-074e25a7769c-bound-sa-token\") pod \"image-registry-68f55d596d-9ppzt\" (UID: \"22cf051d-2e64-45a4-bb98-074e25a7769c\") " pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.799644 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.799586    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:40.847587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:40.847572    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:41.079485 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:41.079407    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68f55d596d-9ppzt"]
Oct 31 13:03:41.082342 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:41.082311    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cf051d_2e64_45a4_bb98_074e25a7769c.slice/crio-ff5177a7aa6cbfadb4ffc697fd6d632b5fe191c48ebc5c40d038e8d9754323f2 WatchSource:0}: Error finding container ff5177a7aa6cbfadb4ffc697fd6d632b5fe191c48ebc5c40d038e8d9754323f2: Status 404 returned error can't find the container with id ff5177a7aa6cbfadb4ffc697fd6d632b5fe191c48ebc5c40d038e8d9754323f2
Oct 31 13:03:41.857279 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:41.857248    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f55d596d-9ppzt" event={"ID":"22cf051d-2e64-45a4-bb98-074e25a7769c","Type":"ContainerStarted","Data":"7f257443a86af069c27a74b8a209944826a02565d511f744c6e834d3b7422b4c"}
Oct 31 13:03:41.857669 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:41.857293    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f55d596d-9ppzt" event={"ID":"22cf051d-2e64-45a4-bb98-074e25a7769c","Type":"ContainerStarted","Data":"ff5177a7aa6cbfadb4ffc697fd6d632b5fe191c48ebc5c40d038e8d9754323f2"}
Oct 31 13:03:41.857669 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:41.857382    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:03:41.883487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:41.883439    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68f55d596d-9ppzt" podStartSLOduration=1.8834264840000001 podStartE2EDuration="1.883426484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:41.882370285 +0000 UTC m=+19.280718910" watchObservedRunningTime="2026-10-31 13:03:41.883426484 +0000 UTC m=+19.281775102"
Oct 31 13:03:43.072626 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.072597    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr"]
Oct 31 13:03:43.076892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.076859    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr"
Oct 31 13:03:43.081787 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.081764    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 13:03:43.081923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.081802    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 13:03:43.081923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.081812    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 13:03:43.082330 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.082293    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 13:03:43.082862 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.082847    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 13:03:43.082964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.082926    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rr7p2\""
Oct 31 13:03:43.089462 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.089442    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr"]
Oct 31 13:03:43.113592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.113563    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:43.172352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.172323    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7fd84c4-0953-473c-a50e-950ffe2febfb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8dqzr\" (UID: \"a7fd84c4-0953-473c-a50e-950ffe2febfb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr"
Oct 31 13:03:43.172455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.172361    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7fd84c4-0953-473c-a50e-950ffe2febfb-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8dqzr\" (UID: \"a7fd84c4-0953-473c-a50e-950ffe2febfb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr"
Oct 31 13:03:43.172497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.172448    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pcv2\" (UniqueName: \"kubernetes.io/projected/a7fd84c4-0953-473c-a50e-950ffe2febfb-kube-api-access-2pcv2\") pod \"prometheus-operator-9cd89b74c-8dqzr\" (UID: \"a7fd84c4-0953-473c-a50e-950ffe2febfb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr"
Oct 31 13:03:43.172497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.172483    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7fd84c4-0953-473c-a50e-950ffe2febfb-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8dqzr\" (UID: \"a7fd84c4-0953-473c-a50e-950ffe2febfb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr"
Oct 31 13:03:43.272904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.272862    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7fd84c4-0953-473c-a50e-950ffe2febfb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8dqzr\" (UID: \"a7fd84c4-0953-473c-a50e-950ffe2febfb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr"
Oct 31 13:03:43.272963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.272912    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7fd84c4-0953-473c-a50e-950ffe2febfb-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8dqzr\" (UID: \"a7fd84c4-0953-473c-a50e-950ffe2febfb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr"
Oct 31 13:03:43.272963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.272944    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pcv2\" (UniqueName: \"kubernetes.io/projected/a7fd84c4-0953-473c-a50e-950ffe2febfb-kube-api-access-2pcv2\") pod \"prometheus-operator-9cd89b74c-8dqzr\" (UID: \"a7fd84c4-0953-473c-a50e-950ffe2febfb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr"
Oct 31 13:03:43.273125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.272963    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7fd84c4-0953-473c-a50e-950ffe2febfb-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8dqzr\" (UID: \"a7fd84c4-0953-473c-a50e-950ffe2febfb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr"
Oct 31 13:03:43.273673 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.273650    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7fd84c4-0953-473c-a50e-950ffe2febfb-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8dqzr\" (UID: \"a7fd84c4-0953-473c-a50e-950ffe2febfb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr"
Oct 31 13:03:43.275511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.275483    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7fd84c4-0953-473c-a50e-950ffe2febfb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8dqzr\" (UID: \"a7fd84c4-0953-473c-a50e-950ffe2febfb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr"
Oct 31 13:03:43.275836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.275818    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7fd84c4-0953-473c-a50e-950ffe2febfb-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8dqzr\" (UID: \"a7fd84c4-0953-473c-a50e-950ffe2febfb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr"
Oct 31 13:03:43.276602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.276581    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:43.285547 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.285527    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pcv2\" (UniqueName: \"kubernetes.io/projected/a7fd84c4-0953-473c-a50e-950ffe2febfb-kube-api-access-2pcv2\") pod \"prometheus-operator-9cd89b74c-8dqzr\" (UID: \"a7fd84c4-0953-473c-a50e-950ffe2febfb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr"
Oct 31 13:03:43.389696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.389642    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr"
Oct 31 13:03:43.659948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.659880    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr"]
Oct 31 13:03:43.663540 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:43.663516    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7fd84c4_0953_473c_a50e_950ffe2febfb.slice/crio-e977b2092141671f650fc27190398a3a0f899fb814b6cb7e70ab6e04552eb8f2 WatchSource:0}: Error finding container e977b2092141671f650fc27190398a3a0f899fb814b6cb7e70ab6e04552eb8f2: Status 404 returned error can't find the container with id e977b2092141671f650fc27190398a3a0f899fb814b6cb7e70ab6e04552eb8f2
Oct 31 13:03:43.815428 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.815399    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:43.864751 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.864719    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr" event={"ID":"a7fd84c4-0953-473c-a50e-950ffe2febfb","Type":"ContainerStarted","Data":"e977b2092141671f650fc27190398a3a0f899fb814b6cb7e70ab6e04552eb8f2"}
Oct 31 13:03:43.977884 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:43.977842    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:45.436422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:45.436350    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:45.874049 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:45.874022    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr" event={"ID":"a7fd84c4-0953-473c-a50e-950ffe2febfb","Type":"ContainerStarted","Data":"59cb71c29edf15a854980f640a37b90c07867429511f54f9682d21219e85f76a"}
Oct 31 13:03:45.874181 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:45.874099    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr" event={"ID":"a7fd84c4-0953-473c-a50e-950ffe2febfb","Type":"ContainerStarted","Data":"f82e6db202d88bd18e9d1b09a1d711f4dfa6d155de7aca09e84408cf0a3ae0b8"}
Oct 31 13:03:45.909123 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:45.909079    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8dqzr" podStartSLOduration=1.513164795 podStartE2EDuration="2.90906726s" totalImagesPullingTime="1.395902465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:45.908614579 +0000 UTC m=+23.306963205" watchObservedRunningTime="2026-10-31 13:03:45.90906726 +0000 UTC m=+23.307415906"
Oct 31 13:03:47.316407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.316376    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:47.510754 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.510729    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"]
Oct 31 13:03:47.514643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.514624    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.531282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.530555    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bsdgs"]
Oct 31 13:03:47.531282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.531145    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 13:03:47.532088 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.532063    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 13:03:47.532088 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.532068    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-p2tqp\""
Oct 31 13:03:47.534029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.534011    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"]
Oct 31 13:03:47.534138 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.534100    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.542725 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.542705    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:47.577269 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.577219    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 13:03:47.577652 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.577637    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 13:03:47.577737 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.577680    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rrldl\""
Oct 31 13:03:47.595113 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.595088    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 13:03:47.595197 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.595139    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 13:03:47.600436 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.600419    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/108cc57c-42a8-43d1-bfbc-2a34de82ba31-metrics-client-ca\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.600483 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.600444    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/108cc57c-42a8-43d1-bfbc-2a34de82ba31-node-exporter-textfile\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.600483 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.600464    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/108cc57c-42a8-43d1-bfbc-2a34de82ba31-node-exporter-accelerators-collector-config\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.600554 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.600488    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b717bd42-13ab-4398-b858-80dc2aa8b887-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.600606 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.600588    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zbm7\" (UniqueName: \"kubernetes.io/projected/b717bd42-13ab-4398-b858-80dc2aa8b887-kube-api-access-7zbm7\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.600676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.600663    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b717bd42-13ab-4398-b858-80dc2aa8b887-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.600710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.600685    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/108cc57c-42a8-43d1-bfbc-2a34de82ba31-root\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.600710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.600704    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/108cc57c-42a8-43d1-bfbc-2a34de82ba31-sys\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.600777 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.600721    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b717bd42-13ab-4398-b858-80dc2aa8b887-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.600824 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.600782    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b717bd42-13ab-4398-b858-80dc2aa8b887-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.600824 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.600802    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/108cc57c-42a8-43d1-bfbc-2a34de82ba31-node-exporter-wtmp\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.600824 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.600818    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b717bd42-13ab-4398-b858-80dc2aa8b887-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.601002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.600882    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/108cc57c-42a8-43d1-bfbc-2a34de82ba31-node-exporter-tls\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.601002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.600976    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzwsh\" (UniqueName: \"kubernetes.io/projected/108cc57c-42a8-43d1-bfbc-2a34de82ba31-kube-api-access-rzwsh\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.601100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.601011    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/108cc57c-42a8-43d1-bfbc-2a34de82ba31-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.702146 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702119    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b717bd42-13ab-4398-b858-80dc2aa8b887-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.702238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702149    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/108cc57c-42a8-43d1-bfbc-2a34de82ba31-root\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.702238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702166    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/108cc57c-42a8-43d1-bfbc-2a34de82ba31-sys\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.702238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702215    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b717bd42-13ab-4398-b858-80dc2aa8b887-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.702392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702249    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/108cc57c-42a8-43d1-bfbc-2a34de82ba31-sys\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.702392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702259    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b717bd42-13ab-4398-b858-80dc2aa8b887-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.702392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702288    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/108cc57c-42a8-43d1-bfbc-2a34de82ba31-node-exporter-wtmp\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.702392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702316    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b717bd42-13ab-4398-b858-80dc2aa8b887-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.702392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702335    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/108cc57c-42a8-43d1-bfbc-2a34de82ba31-node-exporter-tls\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.702392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702360    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzwsh\" (UniqueName: \"kubernetes.io/projected/108cc57c-42a8-43d1-bfbc-2a34de82ba31-kube-api-access-rzwsh\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.702392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702370    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/108cc57c-42a8-43d1-bfbc-2a34de82ba31-node-exporter-wtmp\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.702392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702389    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/108cc57c-42a8-43d1-bfbc-2a34de82ba31-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.702756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702424    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/108cc57c-42a8-43d1-bfbc-2a34de82ba31-metrics-client-ca\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.702756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702448    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/108cc57c-42a8-43d1-bfbc-2a34de82ba31-node-exporter-textfile\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.702756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702478    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/108cc57c-42a8-43d1-bfbc-2a34de82ba31-node-exporter-accelerators-collector-config\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.702756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702517    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b717bd42-13ab-4398-b858-80dc2aa8b887-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.702756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702550    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zbm7\" (UniqueName: \"kubernetes.io/projected/b717bd42-13ab-4398-b858-80dc2aa8b887-kube-api-access-7zbm7\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.702756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702223    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/108cc57c-42a8-43d1-bfbc-2a34de82ba31-root\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.702756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702712    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b717bd42-13ab-4398-b858-80dc2aa8b887-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.703129 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.702920    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/108cc57c-42a8-43d1-bfbc-2a34de82ba31-node-exporter-textfile\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.703129 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.703101    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b717bd42-13ab-4398-b858-80dc2aa8b887-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.703248 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.703226    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b717bd42-13ab-4398-b858-80dc2aa8b887-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.703587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.703564    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/108cc57c-42a8-43d1-bfbc-2a34de82ba31-node-exporter-accelerators-collector-config\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.703713 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.703694    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/108cc57c-42a8-43d1-bfbc-2a34de82ba31-metrics-client-ca\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.704897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.704860    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/108cc57c-42a8-43d1-bfbc-2a34de82ba31-node-exporter-tls\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.705178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.705155    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b717bd42-13ab-4398-b858-80dc2aa8b887-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.705295 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.705278    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b717bd42-13ab-4398-b858-80dc2aa8b887-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.705609 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.705591    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/108cc57c-42a8-43d1-bfbc-2a34de82ba31-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.715490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.715459    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:47.715773 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.715753    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzwsh\" (UniqueName: \"kubernetes.io/projected/108cc57c-42a8-43d1-bfbc-2a34de82ba31-kube-api-access-rzwsh\") pod \"node-exporter-bsdgs\" (UID: \"108cc57c-42a8-43d1-bfbc-2a34de82ba31\") " pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.716025 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.716009    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zbm7\" (UniqueName: \"kubernetes.io/projected/b717bd42-13ab-4398-b858-80dc2aa8b887-kube-api-access-7zbm7\") pod \"kube-state-metrics-5dc8d7d89b-lrsp7\" (UID: \"b717bd42-13ab-4398-b858-80dc2aa8b887\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.829995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.829941    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"
Oct 31 13:03:47.851237 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.851214    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bsdgs"
Oct 31 13:03:47.864196 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:47.864171    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108cc57c_42a8_43d1_bfbc_2a34de82ba31.slice/crio-b47f90fdf8b566f0353c22cebcf0102cba78ce1a95ca374b7a169c096df7efc4 WatchSource:0}: Error finding container b47f90fdf8b566f0353c22cebcf0102cba78ce1a95ca374b7a169c096df7efc4: Status 404 returned error can't find the container with id b47f90fdf8b566f0353c22cebcf0102cba78ce1a95ca374b7a169c096df7efc4
Oct 31 13:03:47.882313 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.882285    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bsdgs" event={"ID":"108cc57c-42a8-43d1-bfbc-2a34de82ba31","Type":"ContainerStarted","Data":"b47f90fdf8b566f0353c22cebcf0102cba78ce1a95ca374b7a169c096df7efc4"}
Oct 31 13:03:47.899208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:47.899140    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:48.079914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:48.079860    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:48.105787 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:48.105712    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7"]
Oct 31 13:03:48.109099 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:48.109077    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb717bd42_13ab_4398_b858_80dc2aa8b887.slice/crio-d70e5c80a7a39b500abef13395f485e1f08968bd6e08a12a7fd5dc3e4d620e57 WatchSource:0}: Error finding container d70e5c80a7a39b500abef13395f485e1f08968bd6e08a12a7fd5dc3e4d620e57: Status 404 returned error can't find the container with id d70e5c80a7a39b500abef13395f485e1f08968bd6e08a12a7fd5dc3e4d620e57
Oct 31 13:03:48.535589 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:48.535555    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:48.738347 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:48.738311    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:48.743651 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:03:48.743622    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:03:48.887611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:48.887533    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7" event={"ID":"b717bd42-13ab-4398-b858-80dc2aa8b887","Type":"ContainerStarted","Data":"d70e5c80a7a39b500abef13395f485e1f08968bd6e08a12a7fd5dc3e4d620e57"}
Oct 31 13:03:49.596831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:49.596251    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:49.767049 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:49.766981    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:49.893296 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:49.893266    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7" event={"ID":"b717bd42-13ab-4398-b858-80dc2aa8b887","Type":"ContainerStarted","Data":"6ca2077d51a91c6b6a47e14a57b06a4f89271e12f85ed1907443fc177f993dd6"}
Oct 31 13:03:49.893466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:49.893329    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7" event={"ID":"b717bd42-13ab-4398-b858-80dc2aa8b887","Type":"ContainerStarted","Data":"09769567f96a9175c6971f46c945e0ed4c89883ff035912e017632de38677918"}
Oct 31 13:03:49.893466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:49.893358    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7" event={"ID":"b717bd42-13ab-4398-b858-80dc2aa8b887","Type":"ContainerStarted","Data":"35f71aeebc5750977790cd121591a46f2ce63376cd7c8f6b1fe0e92f6983d1cb"}
Oct 31 13:03:49.895009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:49.894980    4428 generic.go:356] "Generic (PLEG): container finished" podID="108cc57c-42a8-43d1-bfbc-2a34de82ba31" containerID="cdc6fa52fd8119d3fb17f937a96f50946f0a39d54d67df7fafda6854bdeaeca0" exitCode=0
Oct 31 13:03:49.895121 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:49.895026    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bsdgs" event={"ID":"108cc57c-42a8-43d1-bfbc-2a34de82ba31","Type":"ContainerDied","Data":"cdc6fa52fd8119d3fb17f937a96f50946f0a39d54d67df7fafda6854bdeaeca0"}
Oct 31 13:03:49.918133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:49.918087    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lrsp7" podStartSLOduration=1.534043595 podStartE2EDuration="2.918076569s" totalImagesPullingTime="1.384032974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:49.916668191 +0000 UTC m=+27.315016816" watchObservedRunningTime="2026-10-31 13:03:49.918076569 +0000 UTC m=+27.316425192"
Oct 31 13:03:50.539132 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:50.539098    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:50.692027 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:50.691999    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:50.845081 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:50.844995    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:50.900479 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:50.900448    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bsdgs" event={"ID":"108cc57c-42a8-43d1-bfbc-2a34de82ba31","Type":"ContainerStarted","Data":"20d127fd5f59a1230dc44960282e5e51ff65c559ff685d2aee437e0bf3395f76"}
Oct 31 13:03:50.900627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:50.900499    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bsdgs" event={"ID":"108cc57c-42a8-43d1-bfbc-2a34de82ba31","Type":"ContainerStarted","Data":"5acc8f0e4f00c8c7f8dd0e27c18aef3a99a603c458ce3da30f4ea948f6358b9e"}
Oct 31 13:03:50.925036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:50.924985    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bsdgs" podStartSLOduration=2.631518799 podStartE2EDuration="3.924972358s" totalImagesPullingTime="1.212582343s" totalInitContainerRuntime="80.871216ms" isStatefulPod=true podCreationTimestamp="2026-10-31 13:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:50.92431819 +0000 UTC m=+28.322666815" watchObservedRunningTime="2026-10-31 13:03:50.924972358 +0000 UTC m=+28.323321000"
Oct 31 13:03:51.634709 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:51.634676    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:51.784451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:51.784419    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:51.870038 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:51.870010    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"]
Oct 31 13:03:51.906129 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:51.906048    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"]
Oct 31 13:03:51.906271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:51.906136    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:51.911678 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:51.911658    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 13:03:51.911959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:51.911938    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 13:03:51.912204 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:51.912161    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 13:03:51.912204 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:51.912159    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tl5wf\""
Oct 31 13:03:51.912351 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:51.912201    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-kgj7aurf99qj\""
Oct 31 13:03:51.912351 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:51.912234    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 13:03:52.037188 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.037159    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6592e99-9caf-418f-ac38-9f58bc8f672a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.037334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.037191    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b6592e99-9caf-418f-ac38-9f58bc8f672a-secret-metrics-server-tls\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.037334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.037219    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b6592e99-9caf-418f-ac38-9f58bc8f672a-audit-log\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.037462 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.037358    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b6592e99-9caf-418f-ac38-9f58bc8f672a-metrics-server-audit-profiles\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.037462 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.037396    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5lg8\" (UniqueName: \"kubernetes.io/projected/b6592e99-9caf-418f-ac38-9f58bc8f672a-kube-api-access-v5lg8\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.037578 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.037559    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b6592e99-9caf-418f-ac38-9f58bc8f672a-client-ca-bundle\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.037648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.037632    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b6592e99-9caf-418f-ac38-9f58bc8f672a-secret-metrics-server-client-certs\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.138555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.138522    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b6592e99-9caf-418f-ac38-9f58bc8f672a-client-ca-bundle\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.138672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.138566    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b6592e99-9caf-418f-ac38-9f58bc8f672a-secret-metrics-server-client-certs\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.138725 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.138693    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6592e99-9caf-418f-ac38-9f58bc8f672a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.138792 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.138738    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b6592e99-9caf-418f-ac38-9f58bc8f672a-secret-metrics-server-tls\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.138792 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.138774    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b6592e99-9caf-418f-ac38-9f58bc8f672a-audit-log\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.138929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.138813    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b6592e99-9caf-418f-ac38-9f58bc8f672a-metrics-server-audit-profiles\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.138929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.138844    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5lg8\" (UniqueName: \"kubernetes.io/projected/b6592e99-9caf-418f-ac38-9f58bc8f672a-kube-api-access-v5lg8\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.139260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.139227    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b6592e99-9caf-418f-ac38-9f58bc8f672a-audit-log\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.139494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.139408    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6592e99-9caf-418f-ac38-9f58bc8f672a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.140055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.140025    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b6592e99-9caf-418f-ac38-9f58bc8f672a-metrics-server-audit-profiles\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.141186 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.141153    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b6592e99-9caf-418f-ac38-9f58bc8f672a-secret-metrics-server-client-certs\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.141291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.141230    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b6592e99-9caf-418f-ac38-9f58bc8f672a-secret-metrics-server-tls\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.141371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.141354    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b6592e99-9caf-418f-ac38-9f58bc8f672a-client-ca-bundle\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.151801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.151773    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5lg8\" (UniqueName: \"kubernetes.io/projected/b6592e99-9caf-418f-ac38-9f58bc8f672a-kube-api-access-v5lg8\") pod \"metrics-server-84cd8cdd9d-wh5xd\" (UID: \"b6592e99-9caf-418f-ac38-9f58bc8f672a\") " pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.233697 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.233667    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:03:52.337130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.337103    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:52.401984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.401958    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vl2d9"
Oct 31 13:03:52.491164 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.491089    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:52.532825 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.532796    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"]
Oct 31 13:03:52.536052 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:03:52.536024    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6592e99_9caf_418f_ac38_9f58bc8f672a.slice/crio-fab043cbf146d13d69d6748475b0f2929a70b0984e7ba4a7e62ba2ebc2df4240 WatchSource:0}: Error finding container fab043cbf146d13d69d6748475b0f2929a70b0984e7ba4a7e62ba2ebc2df4240: Status 404 returned error can't find the container with id fab043cbf146d13d69d6748475b0f2929a70b0984e7ba4a7e62ba2ebc2df4240
Oct 31 13:03:52.644019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.643982    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:52.910856 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:52.910776    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd" event={"ID":"b6592e99-9caf-418f-ac38-9f58bc8f672a","Type":"ContainerStarted","Data":"fab043cbf146d13d69d6748475b0f2929a70b0984e7ba4a7e62ba2ebc2df4240"}
Oct 31 13:03:53.847141 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:53.847106    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:54.922294 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:54.922256    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd" event={"ID":"b6592e99-9caf-418f-ac38-9f58bc8f672a","Type":"ContainerStarted","Data":"ebf15232545710374e7c8e060b40c9eca21da93797c2c299e63ba9d0e6b36025"}
Oct 31 13:03:54.944353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:54.944300    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd" podStartSLOduration=1.6705658049999998 podStartE2EDuration="3.944283466s" totalImagesPullingTime="2.273717661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:54.944057651 +0000 UTC m=+32.342406268" watchObservedRunningTime="2026-10-31 13:03:54.944283466 +0000 UTC m=+32.342632093"
Oct 31 13:03:55.530832 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:55.530797    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:55.685764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:55.685733    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:56.533720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:56.533689    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xpskx"
Oct 31 13:03:56.649821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:56.649779    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:03:56.802058 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:03:56.801988    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:02.078915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:02.078842    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:02.247350 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:02.247310    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:02.864499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:02.864472    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68f55d596d-9ppzt"
Oct 31 13:04:03.113181 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:03.113145    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:03.300349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:03.300311    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:03.487362 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:03.487335    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:04.781393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:04.781308    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:04.934878 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:04.934822    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:06.415975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:06.415931    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:06.570657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:06.570616    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:09.248164 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:09.248133    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:09.407906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:09.407849    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:11.003018 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:11.002984    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:11.193946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:11.193914    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:12.234368 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:12.234334    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:04:12.234893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:12.234398    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:04:18.737968 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:04:18.737936    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:04:22.314505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:22.314456    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:22.469396 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:22.469365    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:30.675064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:30.675022    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:30.829423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:30.829397    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:32.240005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:32.239975    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:04:32.243695 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:32.243674    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd"
Oct 31 13:04:34.885331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:34.885285    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:35.065341 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:35.065309    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:37.308775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:37.308741    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:37.468178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:37.468105    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:48.737983 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:04:48.737945    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:04:56.939274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:56.939241    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:04:57.094438 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:04:57.094412    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:05:05.774544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:05:05.774511    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:05:06.400744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:05:06.400707    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:05:06.556128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:05:06.556093    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:05:11.059675 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:05:11.059641    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:05:18.738364 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:05:18.738318    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:05:48.737789 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:05:48.737692    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:05:59.642188 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:05:59.642153    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:05:59.794384 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:05:59.794353    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:06:11.113149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:06:11.113110    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:06:12.146065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:06:12.146028    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:06:12.296567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:06:12.296539    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:06:15.951949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:06:15.951916    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:06:16.111140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:06:16.111107    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:06:16.323827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:06:16.323796    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:06:18.738238 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:06:18.738202    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:06:24.827324 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:06:24.827293    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:06:25.009571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:06:25.009539    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:06:32.410273 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:06:32.410235    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:06:32.567381 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:06:32.567346    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:06:37.319571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:06:37.319534    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:06:37.472970 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:06:37.472941    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:06:45.668584 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:06:45.668520    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:06:48.738803 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:06:48.738769    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:06:51.961805 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:06:51.961774    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:06:52.119448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:06:52.119415    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:06:55.869458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:06:55.869425    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:07:11.165172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:07:11.165086    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:07:18.738299 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:07:18.738263    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:07:48.738483 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:07:48.738439    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:08:11.207793 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:11.207753    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:08:18.738206 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:08:18.738172    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:08:45.521963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:45.521881    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:08:45.727389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:45.727342    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:08:45.904708 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:45.904637    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:08:46.755993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:46.755958    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx"]
Oct 31 13:08:46.759186 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:46.759167    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx"
Oct 31 13:08:46.768752 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:46.768706    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 13:08:46.768943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:46.768927    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 13:08:46.768999 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:46.768952    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-2m2lx\""
Oct 31 13:08:46.779994 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:46.779972    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx"]
Oct 31 13:08:46.876264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:46.876232    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/737ceb06-826e-481e-93b2-29399370caed-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx\" (UID: \"737ceb06-826e-481e-93b2-29399370caed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx"
Oct 31 13:08:46.876425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:46.876267    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p5mn\" (UniqueName: \"kubernetes.io/projected/737ceb06-826e-481e-93b2-29399370caed-kube-api-access-2p5mn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx\" (UID: \"737ceb06-826e-481e-93b2-29399370caed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx"
Oct 31 13:08:46.876499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:46.876481    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/737ceb06-826e-481e-93b2-29399370caed-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx\" (UID: \"737ceb06-826e-481e-93b2-29399370caed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx"
Oct 31 13:08:46.977564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:46.977524    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/737ceb06-826e-481e-93b2-29399370caed-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx\" (UID: \"737ceb06-826e-481e-93b2-29399370caed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx"
Oct 31 13:08:46.977725 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:46.977592    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/737ceb06-826e-481e-93b2-29399370caed-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx\" (UID: \"737ceb06-826e-481e-93b2-29399370caed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx"
Oct 31 13:08:46.977725 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:46.977615    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5mn\" (UniqueName: \"kubernetes.io/projected/737ceb06-826e-481e-93b2-29399370caed-kube-api-access-2p5mn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx\" (UID: \"737ceb06-826e-481e-93b2-29399370caed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx"
Oct 31 13:08:46.978020 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:46.977997    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/737ceb06-826e-481e-93b2-29399370caed-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx\" (UID: \"737ceb06-826e-481e-93b2-29399370caed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx"
Oct 31 13:08:46.978020 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:46.978012    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/737ceb06-826e-481e-93b2-29399370caed-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx\" (UID: \"737ceb06-826e-481e-93b2-29399370caed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx"
Oct 31 13:08:46.990844 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:46.990816    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5mn\" (UniqueName: \"kubernetes.io/projected/737ceb06-826e-481e-93b2-29399370caed-kube-api-access-2p5mn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx\" (UID: \"737ceb06-826e-481e-93b2-29399370caed\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx"
Oct 31 13:08:47.075056 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.074986    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx"
Oct 31 13:08:47.168531 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.168501    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"]
Oct 31 13:08:47.171727 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.171712    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.176528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.176508    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 13:08:47.176760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.176742    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zfm9b\""
Oct 31 13:08:47.178344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.178246    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.178344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.178287    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.178470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.178379    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.178470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.178422    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.178470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.178458    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.178634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.178501    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvnfn\" (UniqueName: \"kubernetes.io/projected/cebc2d15-71f6-4d14-a282-fff9e7c3e272-kube-api-access-vvnfn\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.178634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.178521    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebc2d15-71f6-4d14-a282-fff9e7c3e272-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.179466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.179452    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 13:08:47.179575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.179561    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 13:08:47.180708 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.180692    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 13:08:47.180804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.180728    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 13:08:47.180804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.180757    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 13:08:47.188918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.188896    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"]
Oct 31 13:08:47.234252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.234226    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:08:47.279104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.279079    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.279268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.279121    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.279268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.279165    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvnfn\" (UniqueName: \"kubernetes.io/projected/cebc2d15-71f6-4d14-a282-fff9e7c3e272-kube-api-access-vvnfn\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.279268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.279191    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebc2d15-71f6-4d14-a282-fff9e7c3e272-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.279268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.279245    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.279482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.279278    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.279482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.279329    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.279903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.279843    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.281548 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.281528    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebc2d15-71f6-4d14-a282-fff9e7c3e272-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.281684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.281668    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.282114 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.282098    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.282194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.282141    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.292966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.292936    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvnfn\" (UniqueName: \"kubernetes.io/projected/cebc2d15-71f6-4d14-a282-fff9e7c3e272-kube-api-access-vvnfn\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.297404 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.297379    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-ltcvs\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.355699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.355642    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx"]
Oct 31 13:08:47.358785 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:08:47.358764    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737ceb06_826e_481e_93b2_29399370caed.slice/crio-051581610c34a300cf11233436a46205ac279e7dfcee0224a62a74b6b85a52d7 WatchSource:0}: Error finding container 051581610c34a300cf11233436a46205ac279e7dfcee0224a62a74b6b85a52d7: Status 404 returned error can't find the container with id 051581610c34a300cf11233436a46205ac279e7dfcee0224a62a74b6b85a52d7
Oct 31 13:08:47.360644 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.360624    4428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:08:47.394480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.394451    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:08:47.483324 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.483291    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:47.761004 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.760976    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"]
Oct 31 13:08:47.763806 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:08:47.763783    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebc2d15_71f6_4d14_a282_fff9e7c3e272.slice/crio-76d3e32f7bc5f044673192816452500ccd10bc76d81ea0a6f9d148a451741253 WatchSource:0}: Error finding container 76d3e32f7bc5f044673192816452500ccd10bc76d81ea0a6f9d148a451741253: Status 404 returned error can't find the container with id 76d3e32f7bc5f044673192816452500ccd10bc76d81ea0a6f9d148a451741253
Oct 31 13:08:47.848190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.848165    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs" event={"ID":"cebc2d15-71f6-4d14-a282-fff9e7c3e272","Type":"ContainerStarted","Data":"76d3e32f7bc5f044673192816452500ccd10bc76d81ea0a6f9d148a451741253"}
Oct 31 13:08:47.849281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:47.849248    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx" event={"ID":"737ceb06-826e-481e-93b2-29399370caed","Type":"ContainerStarted","Data":"051581610c34a300cf11233436a46205ac279e7dfcee0224a62a74b6b85a52d7"}
Oct 31 13:08:48.708813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:48.708769    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:08:48.744074 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:08:48.744040    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:08:51.153884 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:51.153560    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:08:51.153884 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:51.153643    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:08:51.749004 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:51.748976    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:08:51.866409 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:51.866372    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs" event={"ID":"cebc2d15-71f6-4d14-a282-fff9e7c3e272","Type":"ContainerStarted","Data":"6781a2d3c303012d68e309ace72155eb4ffca2fec04d53145b55efcbb9821f74"}
Oct 31 13:08:51.866594 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:51.866513    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:51.893234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:51.893184    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs" podStartSLOduration=1.506284345 podStartE2EDuration="4.893167678s" totalImagesPullingTime="3.386883333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:51.891511457 +0000 UTC m=+329.289860088" watchObservedRunningTime="2026-10-31 13:08:51.893167678 +0000 UTC m=+329.291516304"
Oct 31 13:08:51.927331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:51.927295    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:08:52.839915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:52.839857    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:08:52.871437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:52.871408    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:08:52.996519 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:52.996479    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:08:53.156241 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:53.156122    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:08:53.874708 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:53.874679    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx" event={"ID":"737ceb06-826e-481e-93b2-29399370caed","Type":"ContainerStarted","Data":"249108535dd27fdd0546d7b60be2caf982fdddd8bbbc8fcb2ea0de6351cf6a5f"}
Oct 31 13:08:54.878788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:54.878753    4428 generic.go:356] "Generic (PLEG): container finished" podID="737ceb06-826e-481e-93b2-29399370caed" containerID="249108535dd27fdd0546d7b60be2caf982fdddd8bbbc8fcb2ea0de6351cf6a5f" exitCode=0
Oct 31 13:08:54.879183 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:54.878808    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx" event={"ID":"737ceb06-826e-481e-93b2-29399370caed","Type":"ContainerDied","Data":"249108535dd27fdd0546d7b60be2caf982fdddd8bbbc8fcb2ea0de6351cf6a5f"}
Oct 31 13:08:55.419302 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:55.419248    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:08:57.895068 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:57.895035    4428 generic.go:356] "Generic (PLEG): container finished" podID="737ceb06-826e-481e-93b2-29399370caed" containerID="3901ac4e1eab6d1508fddc6717011bcf9e000db2fc75b4e6e10ed594dd4519a8" exitCode=0
Oct 31 13:08:57.895434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:08:57.895118    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx" event={"ID":"737ceb06-826e-481e-93b2-29399370caed","Type":"ContainerDied","Data":"3901ac4e1eab6d1508fddc6717011bcf9e000db2fc75b4e6e10ed594dd4519a8"}
Oct 31 13:09:04.921763 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:04.921728    4428 generic.go:356] "Generic (PLEG): container finished" podID="737ceb06-826e-481e-93b2-29399370caed" containerID="932549affddb469ae55f05a5a64bf158d9f5dc204f486ae1e019fd6e3b364516" exitCode=0
Oct 31 13:09:04.922156 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:04.921824    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx" event={"ID":"737ceb06-826e-481e-93b2-29399370caed","Type":"ContainerDied","Data":"932549affddb469ae55f05a5a64bf158d9f5dc204f486ae1e019fd6e3b364516"}
Oct 31 13:09:06.044922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:06.044900    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx"
Oct 31 13:09:06.124667 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:06.124641    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/737ceb06-826e-481e-93b2-29399370caed-util\" (UniqueName: \"kubernetes.io/empty-dir/737ceb06-826e-481e-93b2-29399370caed-util\") pod \"737ceb06-826e-481e-93b2-29399370caed\" (UID: \"737ceb06-826e-481e-93b2-29399370caed\") "
Oct 31 13:09:06.124829 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:06.124705    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/737ceb06-826e-481e-93b2-29399370caed-bundle\" (UniqueName: \"kubernetes.io/empty-dir/737ceb06-826e-481e-93b2-29399370caed-bundle\") pod \"737ceb06-826e-481e-93b2-29399370caed\" (UID: \"737ceb06-826e-481e-93b2-29399370caed\") "
Oct 31 13:09:06.124829 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:06.124738    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/737ceb06-826e-481e-93b2-29399370caed-kube-api-access-2p5mn\" (UniqueName: \"kubernetes.io/projected/737ceb06-826e-481e-93b2-29399370caed-kube-api-access-2p5mn\") pod \"737ceb06-826e-481e-93b2-29399370caed\" (UID: \"737ceb06-826e-481e-93b2-29399370caed\") "
Oct 31 13:09:06.125365 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:06.125336    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/737ceb06-826e-481e-93b2-29399370caed-bundle" pod "737ceb06-826e-481e-93b2-29399370caed" (UID: "737ceb06-826e-481e-93b2-29399370caed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:06.127047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:06.127026    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737ceb06-826e-481e-93b2-29399370caed-kube-api-access-2p5mn" pod "737ceb06-826e-481e-93b2-29399370caed" (UID: "737ceb06-826e-481e-93b2-29399370caed"). InnerVolumeSpecName "kube-api-access-2p5mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:09:06.128502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:06.128476    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/737ceb06-826e-481e-93b2-29399370caed-util" pod "737ceb06-826e-481e-93b2-29399370caed" (UID: "737ceb06-826e-481e-93b2-29399370caed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:06.225153 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:06.225131    4428 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/737ceb06-826e-481e-93b2-29399370caed-bundle\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:06.225239 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:06.225154    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p5mn\" (UniqueName: \"kubernetes.io/projected/737ceb06-826e-481e-93b2-29399370caed-kube-api-access-2p5mn\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:06.225239 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:06.225164    4428 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/737ceb06-826e-481e-93b2-29399370caed-util\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:06.929435 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:06.929407    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx" event={"ID":"737ceb06-826e-481e-93b2-29399370caed","Type":"ContainerDied","Data":"051581610c34a300cf11233436a46205ac279e7dfcee0224a62a74b6b85a52d7"}
Oct 31 13:09:06.929613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:06.929447    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvqhsx"
Oct 31 13:09:06.929613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:06.929451    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="051581610c34a300cf11233436a46205ac279e7dfcee0224a62a74b6b85a52d7"
Oct 31 13:09:09.429927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:09.429888    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:09.589025 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:09.588987    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:11.275328 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:11.275293    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:18.738557 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:09:18.738523    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:09:19.584305 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:19.584271    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:19.746164 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:19.746114    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:40.917607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:40.917569    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:41.104721 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:41.104693    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:46.030967 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.030935    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x"]
Oct 31 13:09:46.031343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.031188    4428 cpu_manager.go:414] "containerMap: removing container" podUID="737ceb06-826e-481e-93b2-29399370caed" containerName="util"
Oct 31 13:09:46.031343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.031198    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="737ceb06-826e-481e-93b2-29399370caed" containerName="util"
Oct 31 13:09:46.031343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.031214    4428 cpu_manager.go:414] "containerMap: removing container" podUID="737ceb06-826e-481e-93b2-29399370caed" containerName="pull"
Oct 31 13:09:46.031343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.031220    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="737ceb06-826e-481e-93b2-29399370caed" containerName="pull"
Oct 31 13:09:46.031343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.031226    4428 cpu_manager.go:414] "containerMap: removing container" podUID="737ceb06-826e-481e-93b2-29399370caed" containerName="extract"
Oct 31 13:09:46.031343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.031230    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="737ceb06-826e-481e-93b2-29399370caed" containerName="extract"
Oct 31 13:09:46.031343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.031273    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="737ceb06-826e-481e-93b2-29399370caed" containerName="extract"
Oct 31 13:09:46.033390 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.033368    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x"
Oct 31 13:09:46.043981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.043952    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 13:09:46.044177 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.044156    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-tx54x\""
Oct 31 13:09:46.044329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.044314    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 13:09:46.044542 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.044522    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x"]
Oct 31 13:09:46.103185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.103156    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b986c79e-c11a-493c-94df-4fc7125e42e3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x\" (UID: \"b986c79e-c11a-493c-94df-4fc7125e42e3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x"
Oct 31 13:09:46.103336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.103199    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b986c79e-c11a-493c-94df-4fc7125e42e3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x\" (UID: \"b986c79e-c11a-493c-94df-4fc7125e42e3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x"
Oct 31 13:09:46.103336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.103289    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6gxr\" (UniqueName: \"kubernetes.io/projected/b986c79e-c11a-493c-94df-4fc7125e42e3-kube-api-access-z6gxr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x\" (UID: \"b986c79e-c11a-493c-94df-4fc7125e42e3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x"
Oct 31 13:09:46.203729 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.203699    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6gxr\" (UniqueName: \"kubernetes.io/projected/b986c79e-c11a-493c-94df-4fc7125e42e3-kube-api-access-z6gxr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x\" (UID: \"b986c79e-c11a-493c-94df-4fc7125e42e3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x"
Oct 31 13:09:46.203847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.203738    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b986c79e-c11a-493c-94df-4fc7125e42e3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x\" (UID: \"b986c79e-c11a-493c-94df-4fc7125e42e3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x"
Oct 31 13:09:46.203847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.203771    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b986c79e-c11a-493c-94df-4fc7125e42e3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x\" (UID: \"b986c79e-c11a-493c-94df-4fc7125e42e3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x"
Oct 31 13:09:46.204117 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.204101    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b986c79e-c11a-493c-94df-4fc7125e42e3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x\" (UID: \"b986c79e-c11a-493c-94df-4fc7125e42e3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x"
Oct 31 13:09:46.204160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.204141    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b986c79e-c11a-493c-94df-4fc7125e42e3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x\" (UID: \"b986c79e-c11a-493c-94df-4fc7125e42e3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x"
Oct 31 13:09:46.216938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.216912    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6gxr\" (UniqueName: \"kubernetes.io/projected/b986c79e-c11a-493c-94df-4fc7125e42e3-kube-api-access-z6gxr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x\" (UID: \"b986c79e-c11a-493c-94df-4fc7125e42e3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x"
Oct 31 13:09:46.346926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.346840    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x"
Oct 31 13:09:46.417694 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.415907    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:46.574171 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.574134    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:46.623911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:46.620105    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x"]
Oct 31 13:09:46.631814 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:09:46.631775    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb986c79e_c11a_493c_94df_4fc7125e42e3.slice/crio-066e611025b9af0fa8e58ef698eff45c60ac94cb0248fecdf16fb7fcb2ad5e3b WatchSource:0}: Error finding container 066e611025b9af0fa8e58ef698eff45c60ac94cb0248fecdf16fb7fcb2ad5e3b: Status 404 returned error can't find the container with id 066e611025b9af0fa8e58ef698eff45c60ac94cb0248fecdf16fb7fcb2ad5e3b
Oct 31 13:09:47.061038 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.061006    4428 generic.go:356] "Generic (PLEG): container finished" podID="b986c79e-c11a-493c-94df-4fc7125e42e3" containerID="5de6215bb88131a5913e21bf558091555ddd75ac2ae57053851db10daff4d027" exitCode=0
Oct 31 13:09:47.061382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.061046    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x" event={"ID":"b986c79e-c11a-493c-94df-4fc7125e42e3","Type":"ContainerDied","Data":"5de6215bb88131a5913e21bf558091555ddd75ac2ae57053851db10daff4d027"}
Oct 31 13:09:47.061382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.061083    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x" event={"ID":"b986c79e-c11a-493c-94df-4fc7125e42e3","Type":"ContainerStarted","Data":"066e611025b9af0fa8e58ef698eff45c60ac94cb0248fecdf16fb7fcb2ad5e3b"}
Oct 31 13:09:47.481264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.481229    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:47.591722 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.591684    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"]
Oct 31 13:09:47.594672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.594654    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.637470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.637446    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"]
Oct 31 13:09:47.663526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.663500    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 13:09:47.663659 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.663601    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 13:09:47.663659 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.663629    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jj88f\""
Oct 31 13:09:47.663749 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.663712    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 13:09:47.665129 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.665113    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 13:09:47.665215 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.665186    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 13:09:47.716539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.716516    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cd510088-f2ae-4abe-9c80-8c7531915aa0-manifests\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.716631 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.716566    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd510088-f2ae-4abe-9c80-8c7531915aa0-metrics-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.716631 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.716588    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cd510088-f2ae-4abe-9c80-8c7531915aa0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.716726 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.716708    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcbt5\" (UniqueName: \"kubernetes.io/projected/cd510088-f2ae-4abe-9c80-8c7531915aa0-kube-api-access-tcbt5\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.716772 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.716738    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd510088-f2ae-4abe-9c80-8c7531915aa0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.716772 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.716765    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd510088-f2ae-4abe-9c80-8c7531915aa0-tmp\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.791134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.791068    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:47.817186 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.817161    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd510088-f2ae-4abe-9c80-8c7531915aa0-metrics-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.817276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.817189    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cd510088-f2ae-4abe-9c80-8c7531915aa0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.817276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.817221    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcbt5\" (UniqueName: \"kubernetes.io/projected/cd510088-f2ae-4abe-9c80-8c7531915aa0-kube-api-access-tcbt5\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.817276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.817250    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd510088-f2ae-4abe-9c80-8c7531915aa0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.817437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.817275    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd510088-f2ae-4abe-9c80-8c7531915aa0-tmp\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.817437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.817308    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cd510088-f2ae-4abe-9c80-8c7531915aa0-manifests\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.817714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.817686    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cd510088-f2ae-4abe-9c80-8c7531915aa0-manifests\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.819808 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.819782    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd510088-f2ae-4abe-9c80-8c7531915aa0-tmp\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.819959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.819899    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd510088-f2ae-4abe-9c80-8c7531915aa0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.819959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.819911    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cd510088-f2ae-4abe-9c80-8c7531915aa0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.820079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.819966    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd510088-f2ae-4abe-9c80-8c7531915aa0-metrics-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.830525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.830504    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcbt5\" (UniqueName: \"kubernetes.io/projected/cd510088-f2ae-4abe-9c80-8c7531915aa0-kube-api-access-tcbt5\") pod \"opendatahub-operator-controller-manager-7748d49f5-4b2ck\" (UID: \"cd510088-f2ae-4abe-9c80-8c7531915aa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.908970 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.908945    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:47.959601 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:47.959576    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:48.066386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:48.066316    4428 generic.go:356] "Generic (PLEG): container finished" podID="b986c79e-c11a-493c-94df-4fc7125e42e3" containerID="368ebfa2f2816421f3f0b6116f5a49e1f9735361e91d6de30aafe80f932392b6" exitCode=0
Oct 31 13:09:48.066386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:48.066354    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x" event={"ID":"b986c79e-c11a-493c-94df-4fc7125e42e3","Type":"ContainerDied","Data":"368ebfa2f2816421f3f0b6116f5a49e1f9735361e91d6de30aafe80f932392b6"}
Oct 31 13:09:48.124922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:48.124895    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:48.254758 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:48.254710    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"]
Oct 31 13:09:48.256522 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:09:48.256495    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd510088_f2ae_4abe_9c80_8c7531915aa0.slice/crio-15d9822c1dbf7b370a6cf62a69f171d9165b64c3fe1faeb924998614670886c0 WatchSource:0}: Error finding container 15d9822c1dbf7b370a6cf62a69f171d9165b64c3fe1faeb924998614670886c0: Status 404 returned error can't find the container with id 15d9822c1dbf7b370a6cf62a69f171d9165b64c3fe1faeb924998614670886c0
Oct 31 13:09:48.362250 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:48.362173    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:48.540247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:48.540213    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:48.739576 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:09:48.739546    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:09:49.071913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:49.071806    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck" event={"ID":"cd510088-f2ae-4abe-9c80-8c7531915aa0","Type":"ContainerStarted","Data":"15d9822c1dbf7b370a6cf62a69f171d9165b64c3fe1faeb924998614670886c0"}
Oct 31 13:09:49.074889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:49.074846    4428 generic.go:356] "Generic (PLEG): container finished" podID="b986c79e-c11a-493c-94df-4fc7125e42e3" containerID="29db713879b188161be5fed204b10a7eb4c651a21a771b87cd1f0c478401db01" exitCode=0
Oct 31 13:09:49.075003 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:49.074911    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x" event={"ID":"b986c79e-c11a-493c-94df-4fc7125e42e3","Type":"ContainerDied","Data":"29db713879b188161be5fed204b10a7eb4c651a21a771b87cd1f0c478401db01"}
Oct 31 13:09:50.216978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:50.216952    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x"
Oct 31 13:09:50.341202 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:50.340798    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b986c79e-c11a-493c-94df-4fc7125e42e3-util\" (UniqueName: \"kubernetes.io/empty-dir/b986c79e-c11a-493c-94df-4fc7125e42e3-util\") pod \"b986c79e-c11a-493c-94df-4fc7125e42e3\" (UID: \"b986c79e-c11a-493c-94df-4fc7125e42e3\") "
Oct 31 13:09:50.341202 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:50.340855    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b986c79e-c11a-493c-94df-4fc7125e42e3-kube-api-access-z6gxr\" (UniqueName: \"kubernetes.io/projected/b986c79e-c11a-493c-94df-4fc7125e42e3-kube-api-access-z6gxr\") pod \"b986c79e-c11a-493c-94df-4fc7125e42e3\" (UID: \"b986c79e-c11a-493c-94df-4fc7125e42e3\") "
Oct 31 13:09:50.341202 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:50.340934    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b986c79e-c11a-493c-94df-4fc7125e42e3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b986c79e-c11a-493c-94df-4fc7125e42e3-bundle\") pod \"b986c79e-c11a-493c-94df-4fc7125e42e3\" (UID: \"b986c79e-c11a-493c-94df-4fc7125e42e3\") "
Oct 31 13:09:50.341536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:50.341477    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b986c79e-c11a-493c-94df-4fc7125e42e3-bundle" pod "b986c79e-c11a-493c-94df-4fc7125e42e3" (UID: "b986c79e-c11a-493c-94df-4fc7125e42e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:50.343134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:50.343107    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b986c79e-c11a-493c-94df-4fc7125e42e3-kube-api-access-z6gxr" pod "b986c79e-c11a-493c-94df-4fc7125e42e3" (UID: "b986c79e-c11a-493c-94df-4fc7125e42e3"). InnerVolumeSpecName "kube-api-access-z6gxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:09:50.346149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:50.346113    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b986c79e-c11a-493c-94df-4fc7125e42e3-util" pod "b986c79e-c11a-493c-94df-4fc7125e42e3" (UID: "b986c79e-c11a-493c-94df-4fc7125e42e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:09:50.441543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:50.441502    4428 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b986c79e-c11a-493c-94df-4fc7125e42e3-bundle\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:50.441543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:50.441537    4428 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b986c79e-c11a-493c-94df-4fc7125e42e3-util\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:50.441543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:50.441549    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6gxr\" (UniqueName: \"kubernetes.io/projected/b986c79e-c11a-493c-94df-4fc7125e42e3-kube-api-access-z6gxr\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:50.876491 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:50.876452    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:51.084416 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:51.084357    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x"
Oct 31 13:09:51.084600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:51.084351    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nf62x" event={"ID":"b986c79e-c11a-493c-94df-4fc7125e42e3","Type":"ContainerDied","Data":"066e611025b9af0fa8e58ef698eff45c60ac94cb0248fecdf16fb7fcb2ad5e3b"}
Oct 31 13:09:51.084600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:51.084471    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="066e611025b9af0fa8e58ef698eff45c60ac94cb0248fecdf16fb7fcb2ad5e3b"
Oct 31 13:09:51.091815 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:51.091782    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:51.264384 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:51.264340    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:52.089606 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:52.089575    4428 generic.go:356] "Generic (PLEG): container finished" podID="cd510088-f2ae-4abe-9c80-8c7531915aa0" containerID="a6755f2e428f44e984ac36519316f7a4f04a1de22550db11eff33afc4dd2adcb" exitCode=0
Oct 31 13:09:52.089780 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:52.089666    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck" event={"ID":"cd510088-f2ae-4abe-9c80-8c7531915aa0","Type":"ContainerDied","Data":"a6755f2e428f44e984ac36519316f7a4f04a1de22550db11eff33afc4dd2adcb"}
Oct 31 13:09:53.095081 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:53.095052    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck" event={"ID":"cd510088-f2ae-4abe-9c80-8c7531915aa0","Type":"ContainerStarted","Data":"6ced09ab0a2ed4901b43836ff16d6ffe3c702817c89283fb97a5fbbd269bb6ec"}
Oct 31 13:09:53.095445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:53.095194    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:09:53.169085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:53.169031    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck" podStartSLOduration=2.4617731259999998 podStartE2EDuration="6.169016645s" totalImagesPullingTime="3.226399221s" totalInitContainerRuntime="480.844298ms" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:53.168362915 +0000 UTC m=+390.566711540" watchObservedRunningTime="2026-10-31 13:09:53.169016645 +0000 UTC m=+390.567365268"
Oct 31 13:09:55.849351 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:55.849318    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:56.050251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:56.050219    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:09:56.880009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:09:56.879976    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:00.413496 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:00.413463    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:04.101380 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:04.101308    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck"
Oct 31 13:10:05.363376 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:05.363325    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:05.587911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:05.587882    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:05.873540 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:05.873513    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:06.276552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:06.276519    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:06.458401 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:06.458371    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:07.864845 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:07.864816    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:08.151550 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:08.151472    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:08.697657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:08.697623    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:09.023954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:09.023921    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:09.820445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:09.820409    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:10.455276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:10.455245    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:10.889445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:10.889414    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:11.341434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:11.341403    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:13.732263 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:13.732231    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:14.579356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:14.579324    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:14.736033 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:14.736004    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:15.647284 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:15.647244    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:15.805073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:15.805047    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:16.231475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:16.231447    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:18.594164 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.594123    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:18.738511 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:10:18.738474    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:10:18.794810 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.794781    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7485898565-4fxqz"]
Oct 31 13:10:18.795107 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.795092    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b986c79e-c11a-493c-94df-4fc7125e42e3" containerName="extract"
Oct 31 13:10:18.795170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.795108    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b986c79e-c11a-493c-94df-4fc7125e42e3" containerName="extract"
Oct 31 13:10:18.795170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.795127    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b986c79e-c11a-493c-94df-4fc7125e42e3" containerName="util"
Oct 31 13:10:18.795170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.795132    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b986c79e-c11a-493c-94df-4fc7125e42e3" containerName="util"
Oct 31 13:10:18.795170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.795138    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b986c79e-c11a-493c-94df-4fc7125e42e3" containerName="pull"
Oct 31 13:10:18.795170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.795143    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b986c79e-c11a-493c-94df-4fc7125e42e3" containerName="pull"
Oct 31 13:10:18.795325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.795189    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b986c79e-c11a-493c-94df-4fc7125e42e3" containerName="extract"
Oct 31 13:10:18.798117 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.798086    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7485898565-4fxqz"
Oct 31 13:10:18.805251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.804802    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 13:10:18.805251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.804845    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 13:10:18.805251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.804912    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6nszp\""
Oct 31 13:10:18.816141 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.816121    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7485898565-4fxqz"]
Oct 31 13:10:18.820005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.819975    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:18.860298 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.860219    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10d7c0b8-2c3c-4ead-938b-0062041d31ae-tls-certs\") pod \"kube-auth-proxy-7485898565-4fxqz\" (UID: \"10d7c0b8-2c3c-4ead-938b-0062041d31ae\") " pod="openshift-ingress/kube-auth-proxy-7485898565-4fxqz"
Oct 31 13:10:18.860298 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.860249    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10d7c0b8-2c3c-4ead-938b-0062041d31ae-tmp\") pod \"kube-auth-proxy-7485898565-4fxqz\" (UID: \"10d7c0b8-2c3c-4ead-938b-0062041d31ae\") " pod="openshift-ingress/kube-auth-proxy-7485898565-4fxqz"
Oct 31 13:10:18.860298 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.860275    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkk5x\" (UniqueName: \"kubernetes.io/projected/10d7c0b8-2c3c-4ead-938b-0062041d31ae-kube-api-access-lkk5x\") pod \"kube-auth-proxy-7485898565-4fxqz\" (UID: \"10d7c0b8-2c3c-4ead-938b-0062041d31ae\") " pod="openshift-ingress/kube-auth-proxy-7485898565-4fxqz"
Oct 31 13:10:18.961640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.961604    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10d7c0b8-2c3c-4ead-938b-0062041d31ae-tls-certs\") pod \"kube-auth-proxy-7485898565-4fxqz\" (UID: \"10d7c0b8-2c3c-4ead-938b-0062041d31ae\") " pod="openshift-ingress/kube-auth-proxy-7485898565-4fxqz"
Oct 31 13:10:18.961640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.961638    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10d7c0b8-2c3c-4ead-938b-0062041d31ae-tmp\") pod \"kube-auth-proxy-7485898565-4fxqz\" (UID: \"10d7c0b8-2c3c-4ead-938b-0062041d31ae\") " pod="openshift-ingress/kube-auth-proxy-7485898565-4fxqz"
Oct 31 13:10:18.961844 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.961660    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkk5x\" (UniqueName: \"kubernetes.io/projected/10d7c0b8-2c3c-4ead-938b-0062041d31ae-kube-api-access-lkk5x\") pod \"kube-auth-proxy-7485898565-4fxqz\" (UID: \"10d7c0b8-2c3c-4ead-938b-0062041d31ae\") " pod="openshift-ingress/kube-auth-proxy-7485898565-4fxqz"
Oct 31 13:10:18.964008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.963980    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10d7c0b8-2c3c-4ead-938b-0062041d31ae-tmp\") pod \"kube-auth-proxy-7485898565-4fxqz\" (UID: \"10d7c0b8-2c3c-4ead-938b-0062041d31ae\") " pod="openshift-ingress/kube-auth-proxy-7485898565-4fxqz"
Oct 31 13:10:18.964390 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.964367    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10d7c0b8-2c3c-4ead-938b-0062041d31ae-tls-certs\") pod \"kube-auth-proxy-7485898565-4fxqz\" (UID: \"10d7c0b8-2c3c-4ead-938b-0062041d31ae\") " pod="openshift-ingress/kube-auth-proxy-7485898565-4fxqz"
Oct 31 13:10:18.984981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:18.984959    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkk5x\" (UniqueName: \"kubernetes.io/projected/10d7c0b8-2c3c-4ead-938b-0062041d31ae-kube-api-access-lkk5x\") pod \"kube-auth-proxy-7485898565-4fxqz\" (UID: \"10d7c0b8-2c3c-4ead-938b-0062041d31ae\") " pod="openshift-ingress/kube-auth-proxy-7485898565-4fxqz"
Oct 31 13:10:19.033744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:19.033711    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:19.110578 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:19.110497    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7485898565-4fxqz"
Oct 31 13:10:19.214862 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:19.214835    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:19.375414 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:19.375344    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:19.389683 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:19.389656    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7485898565-4fxqz"]
Oct 31 13:10:19.392579 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:10:19.392540    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d7c0b8_2c3c_4ead_938b_0062041d31ae.slice/crio-5c21f79f6cf1227eb0d9ba30145c9ed70aa48461bbd647bb7282a5da6d6d434c WatchSource:0}: Error finding container 5c21f79f6cf1227eb0d9ba30145c9ed70aa48461bbd647bb7282a5da6d6d434c: Status 404 returned error can't find the container with id 5c21f79f6cf1227eb0d9ba30145c9ed70aa48461bbd647bb7282a5da6d6d434c
Oct 31 13:10:20.189380 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:20.189308    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7485898565-4fxqz" event={"ID":"10d7c0b8-2c3c-4ead-938b-0062041d31ae","Type":"ContainerStarted","Data":"5c21f79f6cf1227eb0d9ba30145c9ed70aa48461bbd647bb7282a5da6d6d434c"}
Oct 31 13:10:20.641540 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:20.641505    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:20.911771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:20.911659    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:21.094705 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:21.094674    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:22.199491 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:22.199411    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7485898565-4fxqz" event={"ID":"10d7c0b8-2c3c-4ead-938b-0062041d31ae","Type":"ContainerStarted","Data":"d7e73ddd2c607a99fa4cc5f39154dd07593538bf191aa4d51f4505fde21dfdcc"}
Oct 31 13:10:22.222009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:22.221956    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7485898565-4fxqz" podStartSLOduration=1.868887107 podStartE2EDuration="4.2219441s" totalImagesPullingTime="2.353056993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:22.220045743 +0000 UTC m=+419.618394366" watchObservedRunningTime="2026-10-31 13:10:22.2219441 +0000 UTC m=+419.620292753"
Oct 31 13:10:22.854572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:22.854540    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:23.030380 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:23.030329    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:28.711203 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:28.711162    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:28.905209 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:28.905165    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:33.767245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:33.767199    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:33.923378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:33.923347    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:35.509969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:35.509923    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:35.700352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:35.700320    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:37.513139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:37.513105    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:37.673350 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:37.673308    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:40.133488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:40.133453    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:40.297281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:40.297236    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:42.909487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:42.909459    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:43.064340 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:43.064302    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:47.716394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:47.716365    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:47.880708 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:47.880674    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:48.045005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:48.044912    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:48.202280 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:48.202248    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:48.738956 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:10:48.738921    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:10:56.489640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:56.489590    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:10:56.646400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:10:56.646371    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:11:11.405545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:11.405514    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:11:16.440297 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.440261    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:11:16.505645 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.505597    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-t88kb"]
Oct 31 13:11:16.508641 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.508617    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-t88kb"
Oct 31 13:11:16.516002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.515983    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 13:11:16.516002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.516002    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 13:11:16.521663 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.521642    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-t88kb"]
Oct 31 13:11:16.536820 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.536793    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 31 13:11:16.536951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.536901    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vxjxr\""
Oct 31 13:11:16.553927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.553896    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6brb8"]
Oct 31 13:11:16.557152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.557127    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-6brb8"
Oct 31 13:11:16.572512 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.572489    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6brb8"]
Oct 31 13:11:16.577688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.577671    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 31 13:11:16.577688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.577673    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-bgmr4\""
Oct 31 13:11:16.594172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.594153    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-f5jn6"]
Oct 31 13:11:16.597118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.597103    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-f5jn6"
Oct 31 13:11:16.601359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.601337    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 31 13:11:16.611833 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.611813    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 31 13:11:16.611988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.611950    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-mvl6w\""
Oct 31 13:11:16.613643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.613620    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-f5jn6"]
Oct 31 13:11:16.637214 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.637168    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:11:16.696816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.696737    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74zzg\" (UniqueName: \"kubernetes.io/projected/b027618a-47db-416a-9081-13db8719d2d6-kube-api-access-74zzg\") pod \"odh-model-controller-769875db56-f5jn6\" (UID: \"b027618a-47db-416a-9081-13db8719d2d6\") " pod="opendatahub/odh-model-controller-769875db56-f5jn6"
Oct 31 13:11:16.696816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.696784    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8544\" (UniqueName: \"kubernetes.io/projected/276f7225-018f-4d4d-90a1-3ed0086cafd0-kube-api-access-s8544\") pod \"kserve-controller-manager-776cc66d47-t88kb\" (UID: \"276f7225-018f-4d4d-90a1-3ed0086cafd0\") " pod="opendatahub/kserve-controller-manager-776cc66d47-t88kb"
Oct 31 13:11:16.697055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.696855    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/874303da-8bf8-44e8-bdb9-d9aadf7dd956-tls-certs\") pod \"model-serving-api-75bd89fdfc-6brb8\" (UID: \"874303da-8bf8-44e8-bdb9-d9aadf7dd956\") " pod="opendatahub/model-serving-api-75bd89fdfc-6brb8"
Oct 31 13:11:16.697055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.696895    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b027618a-47db-416a-9081-13db8719d2d6-metrics-certs\") pod \"odh-model-controller-769875db56-f5jn6\" (UID: \"b027618a-47db-416a-9081-13db8719d2d6\") " pod="opendatahub/odh-model-controller-769875db56-f5jn6"
Oct 31 13:11:16.697055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.696927    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9kxt\" (UniqueName: \"kubernetes.io/projected/874303da-8bf8-44e8-bdb9-d9aadf7dd956-kube-api-access-p9kxt\") pod \"model-serving-api-75bd89fdfc-6brb8\" (UID: \"874303da-8bf8-44e8-bdb9-d9aadf7dd956\") " pod="opendatahub/model-serving-api-75bd89fdfc-6brb8"
Oct 31 13:11:16.697055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.696942    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b027618a-47db-416a-9081-13db8719d2d6-cert\") pod \"odh-model-controller-769875db56-f5jn6\" (UID: \"b027618a-47db-416a-9081-13db8719d2d6\") " pod="opendatahub/odh-model-controller-769875db56-f5jn6"
Oct 31 13:11:16.697207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.697054    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/276f7225-018f-4d4d-90a1-3ed0086cafd0-cert\") pod \"kserve-controller-manager-776cc66d47-t88kb\" (UID: \"276f7225-018f-4d4d-90a1-3ed0086cafd0\") " pod="opendatahub/kserve-controller-manager-776cc66d47-t88kb"
Oct 31 13:11:16.797167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.797128    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:11:16.797401 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.797380    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9kxt\" (UniqueName: \"kubernetes.io/projected/874303da-8bf8-44e8-bdb9-d9aadf7dd956-kube-api-access-p9kxt\") pod \"model-serving-api-75bd89fdfc-6brb8\" (UID: \"874303da-8bf8-44e8-bdb9-d9aadf7dd956\") " pod="opendatahub/model-serving-api-75bd89fdfc-6brb8"
Oct 31 13:11:16.797481 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.797414    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b027618a-47db-416a-9081-13db8719d2d6-cert\") pod \"odh-model-controller-769875db56-f5jn6\" (UID: \"b027618a-47db-416a-9081-13db8719d2d6\") " pod="opendatahub/odh-model-controller-769875db56-f5jn6"
Oct 31 13:11:16.797481 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.797460    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/276f7225-018f-4d4d-90a1-3ed0086cafd0-cert\") pod \"kserve-controller-manager-776cc66d47-t88kb\" (UID: \"276f7225-018f-4d4d-90a1-3ed0086cafd0\") " pod="opendatahub/kserve-controller-manager-776cc66d47-t88kb"
Oct 31 13:11:16.797586 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.797508    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74zzg\" (UniqueName: \"kubernetes.io/projected/b027618a-47db-416a-9081-13db8719d2d6-kube-api-access-74zzg\") pod \"odh-model-controller-769875db56-f5jn6\" (UID: \"b027618a-47db-416a-9081-13db8719d2d6\") " pod="opendatahub/odh-model-controller-769875db56-f5jn6"
Oct 31 13:11:16.797586 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.797552    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8544\" (UniqueName: \"kubernetes.io/projected/276f7225-018f-4d4d-90a1-3ed0086cafd0-kube-api-access-s8544\") pod \"kserve-controller-manager-776cc66d47-t88kb\" (UID: \"276f7225-018f-4d4d-90a1-3ed0086cafd0\") " pod="opendatahub/kserve-controller-manager-776cc66d47-t88kb"
Oct 31 13:11:16.797687 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.797590    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/874303da-8bf8-44e8-bdb9-d9aadf7dd956-tls-certs\") pod \"model-serving-api-75bd89fdfc-6brb8\" (UID: \"874303da-8bf8-44e8-bdb9-d9aadf7dd956\") " pod="opendatahub/model-serving-api-75bd89fdfc-6brb8"
Oct 31 13:11:16.797687 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.797615    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b027618a-47db-416a-9081-13db8719d2d6-metrics-certs\") pod \"odh-model-controller-769875db56-f5jn6\" (UID: \"b027618a-47db-416a-9081-13db8719d2d6\") " pod="opendatahub/odh-model-controller-769875db56-f5jn6"
Oct 31 13:11:16.800064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.800038    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b027618a-47db-416a-9081-13db8719d2d6-cert\") pod \"odh-model-controller-769875db56-f5jn6\" (UID: \"b027618a-47db-416a-9081-13db8719d2d6\") " pod="opendatahub/odh-model-controller-769875db56-f5jn6"
Oct 31 13:11:16.800199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.800152    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/276f7225-018f-4d4d-90a1-3ed0086cafd0-cert\") pod \"kserve-controller-manager-776cc66d47-t88kb\" (UID: \"276f7225-018f-4d4d-90a1-3ed0086cafd0\") " pod="opendatahub/kserve-controller-manager-776cc66d47-t88kb"
Oct 31 13:11:16.800256 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.800203    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b027618a-47db-416a-9081-13db8719d2d6-metrics-certs\") pod \"odh-model-controller-769875db56-f5jn6\" (UID: \"b027618a-47db-416a-9081-13db8719d2d6\") " pod="opendatahub/odh-model-controller-769875db56-f5jn6"
Oct 31 13:11:16.800335 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.800318    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/874303da-8bf8-44e8-bdb9-d9aadf7dd956-tls-certs\") pod \"model-serving-api-75bd89fdfc-6brb8\" (UID: \"874303da-8bf8-44e8-bdb9-d9aadf7dd956\") " pod="opendatahub/model-serving-api-75bd89fdfc-6brb8"
Oct 31 13:11:16.811607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.811580    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74zzg\" (UniqueName: \"kubernetes.io/projected/b027618a-47db-416a-9081-13db8719d2d6-kube-api-access-74zzg\") pod \"odh-model-controller-769875db56-f5jn6\" (UID: \"b027618a-47db-416a-9081-13db8719d2d6\") " pod="opendatahub/odh-model-controller-769875db56-f5jn6"
Oct 31 13:11:16.811819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.811799    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8544\" (UniqueName: \"kubernetes.io/projected/276f7225-018f-4d4d-90a1-3ed0086cafd0-kube-api-access-s8544\") pod \"kserve-controller-manager-776cc66d47-t88kb\" (UID: \"276f7225-018f-4d4d-90a1-3ed0086cafd0\") " pod="opendatahub/kserve-controller-manager-776cc66d47-t88kb"
Oct 31 13:11:16.812287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.812263    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9kxt\" (UniqueName: \"kubernetes.io/projected/874303da-8bf8-44e8-bdb9-d9aadf7dd956-kube-api-access-p9kxt\") pod \"model-serving-api-75bd89fdfc-6brb8\" (UID: \"874303da-8bf8-44e8-bdb9-d9aadf7dd956\") " pod="opendatahub/model-serving-api-75bd89fdfc-6brb8"
Oct 31 13:11:16.821005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.820988    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-t88kb"
Oct 31 13:11:16.863067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.863037    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"]
Oct 31 13:11:16.866526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.866506    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:16.869023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.868788    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-6brb8"
Oct 31 13:11:16.884483 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.884454    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"]
Oct 31 13:11:16.910041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.910004    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-f5jn6"
Oct 31 13:11:16.955488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.955352    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:11:16.999589 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.999559    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48fhq\" (UniqueName: \"kubernetes.io/projected/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-kube-api-access-48fhq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:16.999707 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.999630    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:16.999766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.999716    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:16.999766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.999749    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:16.999894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.999782    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:16.999930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.999903    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:16.999930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:16.999923    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:17.100468 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.100433    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:17.100624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.100474    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:17.100624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.100507    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:17.100624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.100548    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:17.100624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.100581    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:17.100624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.100604    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:17.100941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.100644    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48fhq\" (UniqueName: \"kubernetes.io/projected/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-kube-api-access-48fhq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:17.101412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.101389    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:17.102895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.102855    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:17.103057 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.103041    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:17.103506 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.103486    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:17.103599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.103567    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:17.113523 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.113499    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-t88kb"]
Oct 31 13:11:17.114885 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.114849    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:17.117072 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:11:17.117050    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276f7225_018f_4d4d_90a1_3ed0086cafd0.slice/crio-48a27e4866ef875911610b09abe4f59985c281beb56cad49d4bbb3e8aedd55dc WatchSource:0}: Error finding container 48a27e4866ef875911610b09abe4f59985c281beb56cad49d4bbb3e8aedd55dc: Status 404 returned error can't find the container with id 48a27e4866ef875911610b09abe4f59985c281beb56cad49d4bbb3e8aedd55dc
Oct 31 13:11:17.119981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.119956    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48fhq\" (UniqueName: \"kubernetes.io/projected/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-kube-api-access-48fhq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-thzmx\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:17.190095 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.190069    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:17.195228 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.195208    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6brb8"]
Oct 31 13:11:17.196250 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:11:17.196222    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874303da_8bf8_44e8_bdb9_d9aadf7dd956.slice/crio-18b2bdbc179b95a7d7dd46379df7a4e2960ba94fe8de8f03439f3c76f223f336 WatchSource:0}: Error finding container 18b2bdbc179b95a7d7dd46379df7a4e2960ba94fe8de8f03439f3c76f223f336: Status 404 returned error can't find the container with id 18b2bdbc179b95a7d7dd46379df7a4e2960ba94fe8de8f03439f3c76f223f336
Oct 31 13:11:17.223669 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.223644    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-f5jn6"]
Oct 31 13:11:17.226990 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:11:17.226965    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb027618a_47db_416a_9081_13db8719d2d6.slice/crio-1893fe787d9234ced0d43edd62694fb84b0d9395cd9556cb9f44142ec91db4e1 WatchSource:0}: Error finding container 1893fe787d9234ced0d43edd62694fb84b0d9395cd9556cb9f44142ec91db4e1: Status 404 returned error can't find the container with id 1893fe787d9234ced0d43edd62694fb84b0d9395cd9556cb9f44142ec91db4e1
Oct 31 13:11:17.402956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.402915    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-f5jn6" event={"ID":"b027618a-47db-416a-9081-13db8719d2d6","Type":"ContainerStarted","Data":"1893fe787d9234ced0d43edd62694fb84b0d9395cd9556cb9f44142ec91db4e1"}
Oct 31 13:11:17.404013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.403993    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-t88kb" event={"ID":"276f7225-018f-4d4d-90a1-3ed0086cafd0","Type":"ContainerStarted","Data":"48a27e4866ef875911610b09abe4f59985c281beb56cad49d4bbb3e8aedd55dc"}
Oct 31 13:11:17.405124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.405094    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-6brb8" event={"ID":"874303da-8bf8-44e8-bdb9-d9aadf7dd956","Type":"ContainerStarted","Data":"18b2bdbc179b95a7d7dd46379df7a4e2960ba94fe8de8f03439f3c76f223f336"}
Oct 31 13:11:17.479414 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.479233    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"]
Oct 31 13:11:17.481739 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:11:17.481714    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eda1ce0_cc32_4f11_8bd2_c3fe7a42cb92.slice/crio-284454f332aaf6daecf7b90d5d0f3fff8a1a1d869e7f78f037e5e50f656a9cae WatchSource:0}: Error finding container 284454f332aaf6daecf7b90d5d0f3fff8a1a1d869e7f78f037e5e50f656a9cae: Status 404 returned error can't find the container with id 284454f332aaf6daecf7b90d5d0f3fff8a1a1d869e7f78f037e5e50f656a9cae
Oct 31 13:11:17.483789 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.483759    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:11:17.483916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.483832    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:11:17.827046 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:17.826978    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:11:18.016051 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:18.015949    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:11:18.188741 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:18.188616    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:11:18.415849 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:18.415807    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx" event={"ID":"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92","Type":"ContainerStarted","Data":"43408c2e16b08f45fac843b0024d32abd4b45825f7b2639aa7f5bcb6699ca717"}
Oct 31 13:11:18.416116 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:18.415899    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx" event={"ID":"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92","Type":"ContainerStarted","Data":"284454f332aaf6daecf7b90d5d0f3fff8a1a1d869e7f78f037e5e50f656a9cae"}
Oct 31 13:11:18.417731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:18.417705    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:18.419731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:18.419707    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 13:11:18.464539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:18.464480    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx" podStartSLOduration=2.46445862 podStartE2EDuration="2.46445862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:18.463710691 +0000 UTC m=+475.862059322" watchObservedRunningTime="2026-10-31 13:11:18.46445862 +0000 UTC m=+475.862807248"
Oct 31 13:11:18.548077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:18.547265    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"]
Oct 31 13:11:18.548077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:18.547502    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs" podUID="cebc2d15-71f6-4d14-a282-fff9e7c3e272" containerName="discovery" containerID="cri-o://6781a2d3c303012d68e309ace72155eb4ffca2fec04d53145b55efcbb9821f74" gracePeriod=30
Oct 31 13:11:18.768282 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:11:18.768248    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:11:18.862112 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:18.861820    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:11:19.017973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.016885    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-csr-ca-configmap\") pod \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") "
Oct 31 13:11:19.017973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.016976    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cebc2d15-71f6-4d14-a282-fff9e7c3e272-kube-api-access-vvnfn\" (UniqueName: \"kubernetes.io/projected/cebc2d15-71f6-4d14-a282-fff9e7c3e272-kube-api-access-vvnfn\") pod \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") "
Oct 31 13:11:19.017973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.017025    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-kubeconfig\") pod \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") "
Oct 31 13:11:19.017973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.017083    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-cacerts\" (UniqueName: \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-cacerts\") pod \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") "
Oct 31 13:11:19.017973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.017128    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cebc2d15-71f6-4d14-a282-fff9e7c3e272-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebc2d15-71f6-4d14-a282-fff9e7c3e272-local-certs\") pod \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") "
Oct 31 13:11:19.017973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.017177    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-csr-dns-cert\") pod \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") "
Oct 31 13:11:19.017973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.017208    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-token\" (UniqueName: \"kubernetes.io/projected/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-token\") pod \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\" (UID: \"cebc2d15-71f6-4d14-a282-fff9e7c3e272\") "
Oct 31 13:11:19.018473 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.018120    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-csr-ca-configmap" pod "cebc2d15-71f6-4d14-a282-fff9e7c3e272" (UID: "cebc2d15-71f6-4d14-a282-fff9e7c3e272"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:11:19.022328 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.022219    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cebc2d15-71f6-4d14-a282-fff9e7c3e272-kube-api-access-vvnfn" pod "cebc2d15-71f6-4d14-a282-fff9e7c3e272" (UID: "cebc2d15-71f6-4d14-a282-fff9e7c3e272"). InnerVolumeSpecName "kube-api-access-vvnfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:11:19.022463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.022345    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-csr-dns-cert" pod "cebc2d15-71f6-4d14-a282-fff9e7c3e272" (UID: "cebc2d15-71f6-4d14-a282-fff9e7c3e272"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:11:19.023450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.023429    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebc2d15-71f6-4d14-a282-fff9e7c3e272-local-certs" pod "cebc2d15-71f6-4d14-a282-fff9e7c3e272" (UID: "cebc2d15-71f6-4d14-a282-fff9e7c3e272"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:11:19.023850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.023774    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-token" pod "cebc2d15-71f6-4d14-a282-fff9e7c3e272" (UID: "cebc2d15-71f6-4d14-a282-fff9e7c3e272"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:11:19.023966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.023889    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-cacerts" pod "cebc2d15-71f6-4d14-a282-fff9e7c3e272" (UID: "cebc2d15-71f6-4d14-a282-fff9e7c3e272"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:11:19.024156 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.024121    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-kubeconfig" pod "cebc2d15-71f6-4d14-a282-fff9e7c3e272" (UID: "cebc2d15-71f6-4d14-a282-fff9e7c3e272"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:11:19.118813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.118677    4428 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-cacerts\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:19.118813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.118711    4428 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebc2d15-71f6-4d14-a282-fff9e7c3e272-local-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:19.118813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.118727    4428 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-csr-dns-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:19.118813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.118741    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:19.118813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.118755    4428 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-csr-ca-configmap\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:19.118813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.118769    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvnfn\" (UniqueName: \"kubernetes.io/projected/cebc2d15-71f6-4d14-a282-fff9e7c3e272-kube-api-access-vvnfn\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:19.118813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.118782    4428 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cebc2d15-71f6-4d14-a282-fff9e7c3e272-istio-kubeconfig\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:19.424944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.423400    4428 generic.go:356] "Generic (PLEG): container finished" podID="cebc2d15-71f6-4d14-a282-fff9e7c3e272" containerID="6781a2d3c303012d68e309ace72155eb4ffca2fec04d53145b55efcbb9821f74" exitCode=0
Oct 31 13:11:19.424944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.423450    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"
Oct 31 13:11:19.424944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.423481    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs" event={"ID":"cebc2d15-71f6-4d14-a282-fff9e7c3e272","Type":"ContainerDied","Data":"6781a2d3c303012d68e309ace72155eb4ffca2fec04d53145b55efcbb9821f74"}
Oct 31 13:11:19.424944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.423522    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs" event={"ID":"cebc2d15-71f6-4d14-a282-fff9e7c3e272","Type":"ContainerDied","Data":"76d3e32f7bc5f044673192816452500ccd10bc76d81ea0a6f9d148a451741253"}
Oct 31 13:11:19.424944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.423563    4428 scope.go:122] "RemoveContainer" containerID="6781a2d3c303012d68e309ace72155eb4ffca2fec04d53145b55efcbb9821f74"
Oct 31 13:11:19.438524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.438495    4428 scope.go:122] "RemoveContainer" containerID="6781a2d3c303012d68e309ace72155eb4ffca2fec04d53145b55efcbb9821f74"
Oct 31 13:11:19.439014 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:11:19.438967    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6781a2d3c303012d68e309ace72155eb4ffca2fec04d53145b55efcbb9821f74\": container with ID starting with 6781a2d3c303012d68e309ace72155eb4ffca2fec04d53145b55efcbb9821f74 not found: ID does not exist" containerID="6781a2d3c303012d68e309ace72155eb4ffca2fec04d53145b55efcbb9821f74"
Oct 31 13:11:19.439115 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.439029    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6781a2d3c303012d68e309ace72155eb4ffca2fec04d53145b55efcbb9821f74"} err="failed to get container status \"6781a2d3c303012d68e309ace72155eb4ffca2fec04d53145b55efcbb9821f74\": rpc error: code = NotFound desc = could not find container \"6781a2d3c303012d68e309ace72155eb4ffca2fec04d53145b55efcbb9821f74\": container with ID starting with 6781a2d3c303012d68e309ace72155eb4ffca2fec04d53145b55efcbb9821f74 not found: ID does not exist"
Oct 31 13:11:19.462448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.462419    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"]
Oct 31 13:11:19.469008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:19.468985    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-ltcvs"]
Oct 31 13:11:20.431222 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:20.431181    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-t88kb" event={"ID":"276f7225-018f-4d4d-90a1-3ed0086cafd0","Type":"ContainerStarted","Data":"0026517cdcb3746f623e787768887b95c0f01d14c264b6560ab44ed143992587"}
Oct 31 13:11:20.431699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:20.431306    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-t88kb"
Oct 31 13:11:20.434165 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:20.434132    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-6brb8" event={"ID":"874303da-8bf8-44e8-bdb9-d9aadf7dd956","Type":"ContainerStarted","Data":"7d4b8a8091e857115ab4d80317129e69efea2d9778277d4453542597869dc216"}
Oct 31 13:11:20.472988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:20.471452    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-t88kb" podStartSLOduration=1.528173175 podStartE2EDuration="4.471433585s" totalImagesPullingTime="2.94326041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:20.451250524 +0000 UTC m=+477.849599149" watchObservedRunningTime="2026-10-31 13:11:20.471433585 +0000 UTC m=+477.869782221"
Oct 31 13:11:20.472988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:20.472132    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-6brb8" podStartSLOduration=1.742773679 podStartE2EDuration="4.472120541s" totalImagesPullingTime="2.729346862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:20.47077566 +0000 UTC m=+477.869124286" watchObservedRunningTime="2026-10-31 13:11:20.472120541 +0000 UTC m=+477.870469178"
Oct 31 13:11:20.756128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:20.755828    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cebc2d15-71f6-4d14-a282-fff9e7c3e272" path="/var/lib/kubelet/pods/cebc2d15-71f6-4d14-a282-fff9e7c3e272/volumes"
Oct 31 13:11:21.439655 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:21.439614    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-f5jn6" event={"ID":"b027618a-47db-416a-9081-13db8719d2d6","Type":"ContainerStarted","Data":"703dced1374ca0908454fdf151bacc4e0339648fd70a6ecc397522e329868be0"}
Oct 31 13:11:21.440174 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:21.440040    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-6brb8"
Oct 31 13:11:21.460179 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:21.460116    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-f5jn6" podStartSLOduration=1.842661581 podStartE2EDuration="5.460104654s" totalImagesPullingTime="3.617443073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:21.459762768 +0000 UTC m=+478.858111392" watchObservedRunningTime="2026-10-31 13:11:21.460104654 +0000 UTC m=+478.858453317"
Oct 31 13:11:22.444419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:22.444392    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-f5jn6"
Oct 31 13:11:32.448178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:32.448152    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-6brb8"
Oct 31 13:11:33.450935 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:33.450898    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-f5jn6"
Oct 31 13:11:48.739053 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:11:48.738967    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:11:51.444796 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:51.444770    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-t88kb"
Oct 31 13:11:59.881192 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:11:59.881158    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:12:11.442027 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:12:11.441995    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:12:18.738981 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:12:18.738941    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:12:45.682485 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:12:45.682451    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:12:48.338313 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:12:48.338280    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:12:48.494029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:12:48.493994    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:12:48.738626 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:12:48.738521    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:12:57.188822 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:12:57.188782    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:13:11.491928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:13:11.491835    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:13:18.738436 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:13:18.738397    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:13:22.390964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:13:22.390900    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:13:48.738985 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:13:48.738946    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:13:51.922587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:13:51.922524    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:14:11.587692 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:14:11.587648    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:14:18.738526 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:14:18.738492    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:14:28.028888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:14:28.028841    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:14:46.428293 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:14:46.428215    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:14:46.583195 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:14:46.583165    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:14:48.739356 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:14:48.739318    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:15:11.608915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:15:11.608882    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:15:18.739639 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:15:18.739592    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:15:48.740921 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:15:48.740878    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:16:11.656405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:16:11.656281    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:16:18.741403 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:16:18.741366    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:16:48.742490 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:16:48.742450    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:17:11.713938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:17:11.713858    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:17:18.739052 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:17:18.739016    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:17:28.302976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:17:28.302944    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:17:28.463715 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:17:28.463691    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:17:48.738979 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:17:48.738837    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:18:11.774853 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:18:11.774821    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:18:18.738592 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:18:18.738553    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:18:48.738500 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:18:48.738461    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:19:11.886102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:19:11.886035    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:19:18.739495 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:19:18.739461    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:19:48.738010 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:19:48.737972    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:20:11.821835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:11.821782    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:20:18.738599 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:20:18.738558    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:20:48.738510 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:20:48.738430    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:20:54.904771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:54.904734    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:20:57.538925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.538893    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"]
Oct 31 13:20:57.539315 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.539189    4428 cpu_manager.go:414] "containerMap: removing container" podUID="cebc2d15-71f6-4d14-a282-fff9e7c3e272" containerName="discovery"
Oct 31 13:20:57.539315 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.539199    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cebc2d15-71f6-4d14-a282-fff9e7c3e272" containerName="discovery"
Oct 31 13:20:57.539315 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.539244    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="cebc2d15-71f6-4d14-a282-fff9e7c3e272" containerName="discovery"
Oct 31 13:20:57.542101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.542081    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"
Oct 31 13:20:57.547423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.547401    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 13:20:57.547733 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.547712    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rmxc5\""
Oct 31 13:20:57.547733 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.547725    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 13:20:57.547851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.547725    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 13:20:57.552184 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.552163    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"]
Oct 31 13:20:57.576407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.576387    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cfab6c88-d7ae-47df-b616-e13a7e2af77f-config-file\") pod \"limitador-limitador-74dfcfb894-f9pf6\" (UID: \"cfab6c88-d7ae-47df-b616-e13a7e2af77f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"
Oct 31 13:20:57.576535 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.576417    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqzps\" (UniqueName: \"kubernetes.io/projected/cfab6c88-d7ae-47df-b616-e13a7e2af77f-kube-api-access-zqzps\") pod \"limitador-limitador-74dfcfb894-f9pf6\" (UID: \"cfab6c88-d7ae-47df-b616-e13a7e2af77f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"
Oct 31 13:20:57.613141 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.613116    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:20:57.637591 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.637566    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"]
Oct 31 13:20:57.676996 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.676968    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqzps\" (UniqueName: \"kubernetes.io/projected/cfab6c88-d7ae-47df-b616-e13a7e2af77f-kube-api-access-zqzps\") pod \"limitador-limitador-74dfcfb894-f9pf6\" (UID: \"cfab6c88-d7ae-47df-b616-e13a7e2af77f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"
Oct 31 13:20:57.677139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.677066    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cfab6c88-d7ae-47df-b616-e13a7e2af77f-config-file\") pod \"limitador-limitador-74dfcfb894-f9pf6\" (UID: \"cfab6c88-d7ae-47df-b616-e13a7e2af77f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"
Oct 31 13:20:57.677706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.677688    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cfab6c88-d7ae-47df-b616-e13a7e2af77f-config-file\") pod \"limitador-limitador-74dfcfb894-f9pf6\" (UID: \"cfab6c88-d7ae-47df-b616-e13a7e2af77f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"
Oct 31 13:20:57.695294 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.695270    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqzps\" (UniqueName: \"kubernetes.io/projected/cfab6c88-d7ae-47df-b616-e13a7e2af77f-kube-api-access-zqzps\") pod \"limitador-limitador-74dfcfb894-f9pf6\" (UID: \"cfab6c88-d7ae-47df-b616-e13a7e2af77f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"
Oct 31 13:20:57.782611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.782580    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:20:57.855226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.855171    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"
Oct 31 13:20:57.945524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:57.945497    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:20:58.133544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:58.133365    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"]
Oct 31 13:20:58.136242 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:20:58.136220    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfab6c88_d7ae_47df_b616_e13a7e2af77f.slice/crio-e4c9f98f365c815c207a98d102c4a0e66d53443694332ce3482913932e899c78 WatchSource:0}: Error finding container e4c9f98f365c815c207a98d102c4a0e66d53443694332ce3482913932e899c78: Status 404 returned error can't find the container with id e4c9f98f365c815c207a98d102c4a0e66d53443694332ce3482913932e899c78
Oct 31 13:20:58.138168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:58.138151    4428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:20:58.507381 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:58.507349    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:20:58.627219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:58.627188    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6" event={"ID":"cfab6c88-d7ae-47df-b616-e13a7e2af77f","Type":"ContainerStarted","Data":"e4c9f98f365c815c207a98d102c4a0e66d53443694332ce3482913932e899c78"}
Oct 31 13:20:58.662422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:58.662397    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:20:58.769404 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:58.769356    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m4vph"]
Oct 31 13:20:58.773678 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:58.773661    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-m4vph"
Oct 31 13:20:58.780362 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:58.780342    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wp529\""
Oct 31 13:20:58.780521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:58.780499    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m4vph"]
Oct 31 13:20:58.783571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:58.783549    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcklq\" (UniqueName: \"kubernetes.io/projected/5e7cf3af-afa1-4308-b29e-d042c55df1e7-kube-api-access-fcklq\") pod \"authorino-56c6c9ddd5-m4vph\" (UID: \"5e7cf3af-afa1-4308-b29e-d042c55df1e7\") " pod="kuadrant-system/authorino-56c6c9ddd5-m4vph"
Oct 31 13:20:58.884400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:58.884366    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcklq\" (UniqueName: \"kubernetes.io/projected/5e7cf3af-afa1-4308-b29e-d042c55df1e7-kube-api-access-fcklq\") pod \"authorino-56c6c9ddd5-m4vph\" (UID: \"5e7cf3af-afa1-4308-b29e-d042c55df1e7\") " pod="kuadrant-system/authorino-56c6c9ddd5-m4vph"
Oct 31 13:20:58.898153 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:58.898129    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcklq\" (UniqueName: \"kubernetes.io/projected/5e7cf3af-afa1-4308-b29e-d042c55df1e7-kube-api-access-fcklq\") pod \"authorino-56c6c9ddd5-m4vph\" (UID: \"5e7cf3af-afa1-4308-b29e-d042c55df1e7\") " pod="kuadrant-system/authorino-56c6c9ddd5-m4vph"
Oct 31 13:20:59.100813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:59.100740    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-m4vph"
Oct 31 13:20:59.372823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:59.372796    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m4vph"]
Oct 31 13:20:59.375011 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:20:59.374988    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e7cf3af_afa1_4308_b29e_d042c55df1e7.slice/crio-cd16aea33539fb6824502c4be46f6c11d6a9aa2cba6d3f4927c13af7fbd5ad9c WatchSource:0}: Error finding container cd16aea33539fb6824502c4be46f6c11d6a9aa2cba6d3f4927c13af7fbd5ad9c: Status 404 returned error can't find the container with id cd16aea33539fb6824502c4be46f6c11d6a9aa2cba6d3f4927c13af7fbd5ad9c
Oct 31 13:20:59.633708 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:20:59.633637    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-m4vph" event={"ID":"5e7cf3af-afa1-4308-b29e-d042c55df1e7","Type":"ContainerStarted","Data":"cd16aea33539fb6824502c4be46f6c11d6a9aa2cba6d3f4927c13af7fbd5ad9c"}
Oct 31 13:21:01.646253 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:01.646216    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6" event={"ID":"cfab6c88-d7ae-47df-b616-e13a7e2af77f","Type":"ContainerStarted","Data":"97073d41d5e2a22ff591ee452f224fecc3f02242832c43f2e318d4add28d703c"}
Oct 31 13:21:01.646690 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:01.646374    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"
Oct 31 13:21:01.667300 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:01.667257    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6" podStartSLOduration=1.799883276 podStartE2EDuration="4.667244989s" totalImagesPullingTime="2.867361713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:20:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:01.666294233 +0000 UTC m=+1059.064642859" watchObservedRunningTime="2026-10-31 13:21:01.667244989 +0000 UTC m=+1059.065593603"
Oct 31 13:21:02.655060 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:02.655023    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-m4vph" event={"ID":"5e7cf3af-afa1-4308-b29e-d042c55df1e7","Type":"ContainerStarted","Data":"b084deebafabc2a56e50f14ac27a5a69b21de169a0e8532f1634b180831a0ac1"}
Oct 31 13:21:02.673560 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:02.673508    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-m4vph" podStartSLOduration=2.221509992 podStartE2EDuration="4.6734947s" totalImagesPullingTime="2.451984708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:02.671727891 +0000 UTC m=+1060.070076514" watchObservedRunningTime="2026-10-31 13:21:02.6734947 +0000 UTC m=+1060.071843337"
Oct 31 13:21:11.934525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:11.934487    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:21:12.519292 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:12.519264    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"]
Oct 31 13:21:12.519508 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:12.519479    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6" podUID="cfab6c88-d7ae-47df-b616-e13a7e2af77f" containerName="limitador" containerID="cri-o://97073d41d5e2a22ff591ee452f224fecc3f02242832c43f2e318d4add28d703c" gracePeriod=30
Oct 31 13:21:12.520276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:12.520229    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"
Oct 31 13:21:13.060776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:13.060755    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"
Oct 31 13:21:13.094703 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:13.094642    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfab6c88-d7ae-47df-b616-e13a7e2af77f-config-file\" (UniqueName: \"kubernetes.io/configmap/cfab6c88-d7ae-47df-b616-e13a7e2af77f-config-file\") pod \"cfab6c88-d7ae-47df-b616-e13a7e2af77f\" (UID: \"cfab6c88-d7ae-47df-b616-e13a7e2af77f\") "
Oct 31 13:21:13.094703 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:13.094675    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfab6c88-d7ae-47df-b616-e13a7e2af77f-kube-api-access-zqzps\" (UniqueName: \"kubernetes.io/projected/cfab6c88-d7ae-47df-b616-e13a7e2af77f-kube-api-access-zqzps\") pod \"cfab6c88-d7ae-47df-b616-e13a7e2af77f\" (UID: \"cfab6c88-d7ae-47df-b616-e13a7e2af77f\") "
Oct 31 13:21:13.095029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:13.095009    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfab6c88-d7ae-47df-b616-e13a7e2af77f-config-file" pod "cfab6c88-d7ae-47df-b616-e13a7e2af77f" (UID: "cfab6c88-d7ae-47df-b616-e13a7e2af77f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:21:13.096882 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:13.096843    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfab6c88-d7ae-47df-b616-e13a7e2af77f-kube-api-access-zqzps" pod "cfab6c88-d7ae-47df-b616-e13a7e2af77f" (UID: "cfab6c88-d7ae-47df-b616-e13a7e2af77f"). InnerVolumeSpecName "kube-api-access-zqzps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:21:13.195474 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:13.195451    4428 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cfab6c88-d7ae-47df-b616-e13a7e2af77f-config-file\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:21:13.195474 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:13.195475    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqzps\" (UniqueName: \"kubernetes.io/projected/cfab6c88-d7ae-47df-b616-e13a7e2af77f-kube-api-access-zqzps\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:21:13.700544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:13.700518    4428 generic.go:356] "Generic (PLEG): container finished" podID="cfab6c88-d7ae-47df-b616-e13a7e2af77f" containerID="97073d41d5e2a22ff591ee452f224fecc3f02242832c43f2e318d4add28d703c" exitCode=0
Oct 31 13:21:13.700698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:13.700566    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"
Oct 31 13:21:13.700698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:13.700610    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6" event={"ID":"cfab6c88-d7ae-47df-b616-e13a7e2af77f","Type":"ContainerDied","Data":"97073d41d5e2a22ff591ee452f224fecc3f02242832c43f2e318d4add28d703c"}
Oct 31 13:21:13.700698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:13.700662    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-f9pf6" event={"ID":"cfab6c88-d7ae-47df-b616-e13a7e2af77f","Type":"ContainerDied","Data":"e4c9f98f365c815c207a98d102c4a0e66d53443694332ce3482913932e899c78"}
Oct 31 13:21:13.700698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:13.700685    4428 scope.go:122] "RemoveContainer" containerID="97073d41d5e2a22ff591ee452f224fecc3f02242832c43f2e318d4add28d703c"
Oct 31 13:21:13.708984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:13.708968    4428 scope.go:122] "RemoveContainer" containerID="97073d41d5e2a22ff591ee452f224fecc3f02242832c43f2e318d4add28d703c"
Oct 31 13:21:13.709239 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:21:13.709223    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97073d41d5e2a22ff591ee452f224fecc3f02242832c43f2e318d4add28d703c\": container with ID starting with 97073d41d5e2a22ff591ee452f224fecc3f02242832c43f2e318d4add28d703c not found: ID does not exist" containerID="97073d41d5e2a22ff591ee452f224fecc3f02242832c43f2e318d4add28d703c"
Oct 31 13:21:13.709277 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:13.709248    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97073d41d5e2a22ff591ee452f224fecc3f02242832c43f2e318d4add28d703c"} err="failed to get container status \"97073d41d5e2a22ff591ee452f224fecc3f02242832c43f2e318d4add28d703c\": rpc error: code = NotFound desc = could not find container \"97073d41d5e2a22ff591ee452f224fecc3f02242832c43f2e318d4add28d703c\": container with ID starting with 97073d41d5e2a22ff591ee452f224fecc3f02242832c43f2e318d4add28d703c not found: ID does not exist"
Oct 31 13:21:13.722383 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:13.722364    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"]
Oct 31 13:21:13.728691 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:13.728669    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f9pf6"]
Oct 31 13:21:14.752163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:14.752132    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfab6c88-d7ae-47df-b616-e13a7e2af77f" path="/var/lib/kubelet/pods/cfab6c88-d7ae-47df-b616-e13a7e2af77f/volumes"
Oct 31 13:21:18.187025 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.186995    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:21:18.352555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.352526    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:21:18.739201 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:21:18.739160    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:21:18.782155 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.782118    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-h7fm4"]
Oct 31 13:21:18.782409 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.782398    4428 cpu_manager.go:414] "containerMap: removing container" podUID="cfab6c88-d7ae-47df-b616-e13a7e2af77f" containerName="limitador"
Oct 31 13:21:18.782451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.782410    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfab6c88-d7ae-47df-b616-e13a7e2af77f" containerName="limitador"
Oct 31 13:21:18.782484 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.782456    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfab6c88-d7ae-47df-b616-e13a7e2af77f" containerName="limitador"
Oct 31 13:21:18.786645 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.786626    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h7fm4"
Oct 31 13:21:18.795065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.795045    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 31 13:21:18.795794 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.795772    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:21:18.795923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.795903    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-g5bjs\""
Oct 31 13:21:18.796075 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.796050    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:21:18.796264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.796246    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-h7fm4"]
Oct 31 13:21:18.833485 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.833461    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/90cd394c-0002-4feb-aa67-2ddc2bfc9411-data\") pod \"postgres-7cdbf6dc44-h7fm4\" (UID: \"90cd394c-0002-4feb-aa67-2ddc2bfc9411\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h7fm4"
Oct 31 13:21:18.833643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.833512    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpg9m\" (UniqueName: \"kubernetes.io/projected/90cd394c-0002-4feb-aa67-2ddc2bfc9411-kube-api-access-lpg9m\") pod \"postgres-7cdbf6dc44-h7fm4\" (UID: \"90cd394c-0002-4feb-aa67-2ddc2bfc9411\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h7fm4"
Oct 31 13:21:18.934731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.934701    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/90cd394c-0002-4feb-aa67-2ddc2bfc9411-data\") pod \"postgres-7cdbf6dc44-h7fm4\" (UID: \"90cd394c-0002-4feb-aa67-2ddc2bfc9411\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h7fm4"
Oct 31 13:21:18.934887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.934745    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpg9m\" (UniqueName: \"kubernetes.io/projected/90cd394c-0002-4feb-aa67-2ddc2bfc9411-kube-api-access-lpg9m\") pod \"postgres-7cdbf6dc44-h7fm4\" (UID: \"90cd394c-0002-4feb-aa67-2ddc2bfc9411\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h7fm4"
Oct 31 13:21:18.935153 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.935136    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/90cd394c-0002-4feb-aa67-2ddc2bfc9411-data\") pod \"postgres-7cdbf6dc44-h7fm4\" (UID: \"90cd394c-0002-4feb-aa67-2ddc2bfc9411\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h7fm4"
Oct 31 13:21:18.945267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.945245    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:21:18.948102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:18.948086    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpg9m\" (UniqueName: \"kubernetes.io/projected/90cd394c-0002-4feb-aa67-2ddc2bfc9411-kube-api-access-lpg9m\") pod \"postgres-7cdbf6dc44-h7fm4\" (UID: \"90cd394c-0002-4feb-aa67-2ddc2bfc9411\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h7fm4"
Oct 31 13:21:19.099547 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:19.099478    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h7fm4"
Oct 31 13:21:19.371776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:19.371742    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-h7fm4"]
Oct 31 13:21:19.374595 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:21:19.374569    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cd394c_0002_4feb_aa67_2ddc2bfc9411.slice/crio-7f6f0474695342158accd31006e741f04d3e0b5ca9caa7110d2b31d53fd5c6e1 WatchSource:0}: Error finding container 7f6f0474695342158accd31006e741f04d3e0b5ca9caa7110d2b31d53fd5c6e1: Status 404 returned error can't find the container with id 7f6f0474695342158accd31006e741f04d3e0b5ca9caa7110d2b31d53fd5c6e1
Oct 31 13:21:19.725142 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:19.725115    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h7fm4" event={"ID":"90cd394c-0002-4feb-aa67-2ddc2bfc9411","Type":"ContainerStarted","Data":"7f6f0474695342158accd31006e741f04d3e0b5ca9caa7110d2b31d53fd5c6e1"}
Oct 31 13:21:25.327484 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:25.327464    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 31 13:21:25.752227 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:25.752196    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h7fm4" event={"ID":"90cd394c-0002-4feb-aa67-2ddc2bfc9411","Type":"ContainerStarted","Data":"21c88e05eb6c1cdff296c7db5c7948b387df43216506fcf118b95f9eed51e9c1"}
Oct 31 13:21:25.752388 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:25.752256    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h7fm4"
Oct 31 13:21:25.772832 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:25.772787    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h7fm4" podStartSLOduration=1.8246293599999999 podStartE2EDuration="7.772774723s" totalImagesPullingTime="5.948145363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:25.770924361 +0000 UTC m=+1083.169272987" watchObservedRunningTime="2026-10-31 13:21:25.772774723 +0000 UTC m=+1083.171123346"
Oct 31 13:21:31.785452 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:31.785426    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h7fm4"
Oct 31 13:21:34.096860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:34.096828    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:21:34.255370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:34.255343    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:21:38.582396 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:38.582359    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:21:38.738289 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:38.738258    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:21:38.898406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:38.898332    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:21:42.258951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:42.258918    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:21:42.458841 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:42.458787    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:21:43.826552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:43.826517    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"]
Oct 31 13:21:43.830009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:43.829984    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"
Oct 31 13:21:43.836212 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:43.836188    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 31 13:21:43.836917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:43.836899    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-sbr8r\""
Oct 31 13:21:43.837060 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:43.836937    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 31 13:21:43.837060 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:43.836958    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 31 13:21:43.842287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:43.842256    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"]
Oct 31 13:21:43.922203 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:43.922177    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25cbee37-655e-42f3-a41c-36ff91af938e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vkln9\" (UID: \"25cbee37-655e-42f3-a41c-36ff91af938e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"
Oct 31 13:21:43.922333 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:43.922209    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25cbee37-655e-42f3-a41c-36ff91af938e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vkln9\" (UID: \"25cbee37-655e-42f3-a41c-36ff91af938e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"
Oct 31 13:21:43.922333 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:43.922238    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/25cbee37-655e-42f3-a41c-36ff91af938e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vkln9\" (UID: \"25cbee37-655e-42f3-a41c-36ff91af938e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"
Oct 31 13:21:43.922404 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:43.922372    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv7pv\" (UniqueName: \"kubernetes.io/projected/25cbee37-655e-42f3-a41c-36ff91af938e-kube-api-access-fv7pv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vkln9\" (UID: \"25cbee37-655e-42f3-a41c-36ff91af938e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"
Oct 31 13:21:43.946518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:43.946484    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:21:44.022718 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:44.022693    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv7pv\" (UniqueName: \"kubernetes.io/projected/25cbee37-655e-42f3-a41c-36ff91af938e-kube-api-access-fv7pv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vkln9\" (UID: \"25cbee37-655e-42f3-a41c-36ff91af938e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"
Oct 31 13:21:44.022823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:44.022760    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25cbee37-655e-42f3-a41c-36ff91af938e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vkln9\" (UID: \"25cbee37-655e-42f3-a41c-36ff91af938e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"
Oct 31 13:21:44.022823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:44.022791    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25cbee37-655e-42f3-a41c-36ff91af938e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vkln9\" (UID: \"25cbee37-655e-42f3-a41c-36ff91af938e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"
Oct 31 13:21:44.022940 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:44.022821    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/25cbee37-655e-42f3-a41c-36ff91af938e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vkln9\" (UID: \"25cbee37-655e-42f3-a41c-36ff91af938e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"
Oct 31 13:21:44.023599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:44.023527    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25cbee37-655e-42f3-a41c-36ff91af938e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vkln9\" (UID: \"25cbee37-655e-42f3-a41c-36ff91af938e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"
Oct 31 13:21:44.023688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:44.023656    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/25cbee37-655e-42f3-a41c-36ff91af938e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vkln9\" (UID: \"25cbee37-655e-42f3-a41c-36ff91af938e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"
Oct 31 13:21:44.025306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:44.025288    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25cbee37-655e-42f3-a41c-36ff91af938e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vkln9\" (UID: \"25cbee37-655e-42f3-a41c-36ff91af938e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"
Oct 31 13:21:44.036389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:44.036367    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv7pv\" (UniqueName: \"kubernetes.io/projected/25cbee37-655e-42f3-a41c-36ff91af938e-kube-api-access-fv7pv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vkln9\" (UID: \"25cbee37-655e-42f3-a41c-36ff91af938e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"
Oct 31 13:21:44.102053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:44.101987    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:21:44.147160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:44.147139    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"
Oct 31 13:21:44.422089 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:44.422066    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"]
Oct 31 13:21:44.424514 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:21:44.424494    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cbee37_655e_42f3_a41c_36ff91af938e.slice/crio-ccf0f457ed80ee90a57af165006caf7a03ef2bc33e43515177b73b25341a0c50 WatchSource:0}: Error finding container ccf0f457ed80ee90a57af165006caf7a03ef2bc33e43515177b73b25341a0c50: Status 404 returned error can't find the container with id ccf0f457ed80ee90a57af165006caf7a03ef2bc33e43515177b73b25341a0c50
Oct 31 13:21:44.823627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:44.823590    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9" event={"ID":"25cbee37-655e-42f3-a41c-36ff91af938e","Type":"ContainerStarted","Data":"ccf0f457ed80ee90a57af165006caf7a03ef2bc33e43515177b73b25341a0c50"}
Oct 31 13:21:48.752089 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:21:48.752046    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:21:55.878662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:55.878588    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9" event={"ID":"25cbee37-655e-42f3-a41c-36ff91af938e","Type":"ContainerStarted","Data":"1bf092f528e7977669ceea9e055a68466b5a9d6c0f18dabcbf6ae8cfc0744ed9"}
Oct 31 13:21:55.879078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:55.878733    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"
Oct 31 13:21:55.899632 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:21:55.899590    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9" podStartSLOduration=1.7123074219999999 podStartE2EDuration="12.899578315s" totalImagesPullingTime="11.187270893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:55.898372915 +0000 UTC m=+1113.296721539" watchObservedRunningTime="2026-10-31 13:21:55.899578315 +0000 UTC m=+1113.297926932"
Oct 31 13:22:06.888479 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:06.888404    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vkln9"
Oct 31 13:22:11.991020 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:11.990988    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:22:12.971969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:12.971920    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:22:13.132412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:13.132387    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:22:14.281703 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:14.281663    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:22:14.443492 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:14.443466    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:22:18.743317 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:22:18.743278    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:22:23.196060 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:23.196023    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:22:23.403050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:23.403021    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:22:23.534297 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:23.534272    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fbv7g"]
Oct 31 13:22:23.780810 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:23.780777    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fbv7g"]
Oct 31 13:22:23.780976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:23.780911    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-fbv7g"
Oct 31 13:22:23.788423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:23.788401    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 31 13:22:23.796336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:23.796316    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fbv7g"]
Oct 31 13:22:23.801383 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:22:23.801359    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-pxn9n tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-pxn9n tls-cert]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-fbv7g" podUID="03065bbf-072a-4d57-a919-c77a74478237"
Oct 31 13:22:23.826499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:23.826468    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-575b5b59b7-6gpnj"]
Oct 31 13:22:23.841251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:23.841227    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-575b5b59b7-6gpnj"]
Oct 31 13:22:23.841405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:23.841329    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-575b5b59b7-6gpnj"
Oct 31 13:22:23.926829 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:23.926804    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxn9n\" (UniqueName: \"kubernetes.io/projected/03065bbf-072a-4d57-a919-c77a74478237-kube-api-access-pxn9n\") pod \"authorino-6889cbd8c4-fbv7g\" (UID: \"03065bbf-072a-4d57-a919-c77a74478237\") " pod="kuadrant-system/authorino-6889cbd8c4-fbv7g"
Oct 31 13:22:23.926966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:23.926841    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/03065bbf-072a-4d57-a919-c77a74478237-tls-cert\") pod \"authorino-6889cbd8c4-fbv7g\" (UID: \"03065bbf-072a-4d57-a919-c77a74478237\") " pod="kuadrant-system/authorino-6889cbd8c4-fbv7g"
Oct 31 13:22:23.988069 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:23.988044    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-fbv7g"
Oct 31 13:22:24.005586 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.005567    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-fbv7g"
Oct 31 13:22:24.027941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.027916    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/03065bbf-072a-4d57-a919-c77a74478237-tls-cert\") pod \"authorino-6889cbd8c4-fbv7g\" (UID: \"03065bbf-072a-4d57-a919-c77a74478237\") " pod="kuadrant-system/authorino-6889cbd8c4-fbv7g"
Oct 31 13:22:24.028079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.027991    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jcrc\" (UniqueName: \"kubernetes.io/projected/055782a8-f2d0-49a3-b475-ee6e8f09fcd3-kube-api-access-4jcrc\") pod \"authorino-575b5b59b7-6gpnj\" (UID: \"055782a8-f2d0-49a3-b475-ee6e8f09fcd3\") " pod="kuadrant-system/authorino-575b5b59b7-6gpnj"
Oct 31 13:22:24.028079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.028052    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/055782a8-f2d0-49a3-b475-ee6e8f09fcd3-tls-cert\") pod \"authorino-575b5b59b7-6gpnj\" (UID: \"055782a8-f2d0-49a3-b475-ee6e8f09fcd3\") " pod="kuadrant-system/authorino-575b5b59b7-6gpnj"
Oct 31 13:22:24.028200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.028158    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxn9n\" (UniqueName: \"kubernetes.io/projected/03065bbf-072a-4d57-a919-c77a74478237-kube-api-access-pxn9n\") pod \"authorino-6889cbd8c4-fbv7g\" (UID: \"03065bbf-072a-4d57-a919-c77a74478237\") " pod="kuadrant-system/authorino-6889cbd8c4-fbv7g"
Oct 31 13:22:24.030463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.030440    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/03065bbf-072a-4d57-a919-c77a74478237-tls-cert\") pod \"authorino-6889cbd8c4-fbv7g\" (UID: \"03065bbf-072a-4d57-a919-c77a74478237\") " pod="kuadrant-system/authorino-6889cbd8c4-fbv7g"
Oct 31 13:22:24.041184 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.041137    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxn9n\" (UniqueName: \"kubernetes.io/projected/03065bbf-072a-4d57-a919-c77a74478237-kube-api-access-pxn9n\") pod \"authorino-6889cbd8c4-fbv7g\" (UID: \"03065bbf-072a-4d57-a919-c77a74478237\") " pod="kuadrant-system/authorino-6889cbd8c4-fbv7g"
Oct 31 13:22:24.128916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.128892    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/055782a8-f2d0-49a3-b475-ee6e8f09fcd3-tls-cert\") pod \"authorino-575b5b59b7-6gpnj\" (UID: \"055782a8-f2d0-49a3-b475-ee6e8f09fcd3\") " pod="kuadrant-system/authorino-575b5b59b7-6gpnj"
Oct 31 13:22:24.129016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.128960    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jcrc\" (UniqueName: \"kubernetes.io/projected/055782a8-f2d0-49a3-b475-ee6e8f09fcd3-kube-api-access-4jcrc\") pod \"authorino-575b5b59b7-6gpnj\" (UID: \"055782a8-f2d0-49a3-b475-ee6e8f09fcd3\") " pod="kuadrant-system/authorino-575b5b59b7-6gpnj"
Oct 31 13:22:24.131261 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.131238    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/055782a8-f2d0-49a3-b475-ee6e8f09fcd3-tls-cert\") pod \"authorino-575b5b59b7-6gpnj\" (UID: \"055782a8-f2d0-49a3-b475-ee6e8f09fcd3\") " pod="kuadrant-system/authorino-575b5b59b7-6gpnj"
Oct 31 13:22:24.141848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.141830    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jcrc\" (UniqueName: \"kubernetes.io/projected/055782a8-f2d0-49a3-b475-ee6e8f09fcd3-kube-api-access-4jcrc\") pod \"authorino-575b5b59b7-6gpnj\" (UID: \"055782a8-f2d0-49a3-b475-ee6e8f09fcd3\") " pod="kuadrant-system/authorino-575b5b59b7-6gpnj"
Oct 31 13:22:24.169680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.169659    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-575b5b59b7-6gpnj"
Oct 31 13:22:24.229211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.229191    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03065bbf-072a-4d57-a919-c77a74478237-kube-api-access-pxn9n\" (UniqueName: \"kubernetes.io/projected/03065bbf-072a-4d57-a919-c77a74478237-kube-api-access-pxn9n\") pod \"03065bbf-072a-4d57-a919-c77a74478237\" (UID: \"03065bbf-072a-4d57-a919-c77a74478237\") "
Oct 31 13:22:24.229511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.229263    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03065bbf-072a-4d57-a919-c77a74478237-tls-cert\" (UniqueName: \"kubernetes.io/secret/03065bbf-072a-4d57-a919-c77a74478237-tls-cert\") pod \"03065bbf-072a-4d57-a919-c77a74478237\" (UID: \"03065bbf-072a-4d57-a919-c77a74478237\") "
Oct 31 13:22:24.231381 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.231359    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03065bbf-072a-4d57-a919-c77a74478237-tls-cert" pod "03065bbf-072a-4d57-a919-c77a74478237" (UID: "03065bbf-072a-4d57-a919-c77a74478237"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:22:24.231476 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.231428    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03065bbf-072a-4d57-a919-c77a74478237-kube-api-access-pxn9n" pod "03065bbf-072a-4d57-a919-c77a74478237" (UID: "03065bbf-072a-4d57-a919-c77a74478237"). InnerVolumeSpecName "kube-api-access-pxn9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:22:24.330472 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.330414    4428 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/03065bbf-072a-4d57-a919-c77a74478237-tls-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:22:24.330472 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.330441    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxn9n\" (UniqueName: \"kubernetes.io/projected/03065bbf-072a-4d57-a919-c77a74478237-kube-api-access-pxn9n\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:22:24.483857 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.483828    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-575b5b59b7-6gpnj"]
Oct 31 13:22:24.487131 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:22:24.487104    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod055782a8_f2d0_49a3_b475_ee6e8f09fcd3.slice/crio-e707fd924205611593a77bea9678fcfb430b3b02e68fb91d4c070838e16ce0e4 WatchSource:0}: Error finding container e707fd924205611593a77bea9678fcfb430b3b02e68fb91d4c070838e16ce0e4: Status 404 returned error can't find the container with id e707fd924205611593a77bea9678fcfb430b3b02e68fb91d4c070838e16ce0e4
Oct 31 13:22:24.992705 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.992668    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-575b5b59b7-6gpnj" event={"ID":"055782a8-f2d0-49a3-b475-ee6e8f09fcd3","Type":"ContainerStarted","Data":"e707fd924205611593a77bea9678fcfb430b3b02e68fb91d4c070838e16ce0e4"}
Oct 31 13:22:24.992852 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:24.992687    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-fbv7g"
Oct 31 13:22:25.026774 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:25.026715    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fbv7g"]
Oct 31 13:22:25.034910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:25.034884    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fbv7g"]
Oct 31 13:22:25.999398 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:25.999360    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-575b5b59b7-6gpnj" event={"ID":"055782a8-f2d0-49a3-b475-ee6e8f09fcd3","Type":"ContainerStarted","Data":"61d967d8d70a3b15833c44cde421b00f1158505566579347f737bb1c339aa78d"}
Oct 31 13:22:26.050038 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:26.049975    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-575b5b59b7-6gpnj" podStartSLOduration=2.103927194 podStartE2EDuration="3.049955327s" totalImagesPullingTime="946.028133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:22:26.020550758 +0000 UTC m=+1143.418899389" watchObservedRunningTime="2026-10-31 13:22:26.049955327 +0000 UTC m=+1143.448303949"
Oct 31 13:22:26.050991 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:26.050958    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m4vph"]
Oct 31 13:22:26.051203 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:26.051178    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-m4vph" podUID="5e7cf3af-afa1-4308-b29e-d042c55df1e7" containerName="authorino" containerID="cri-o://b084deebafabc2a56e50f14ac27a5a69b21de169a0e8532f1634b180831a0ac1" gracePeriod=30
Oct 31 13:22:26.357287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:26.357262    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-m4vph"
Oct 31 13:22:26.445519 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:26.445488    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e7cf3af-afa1-4308-b29e-d042c55df1e7-kube-api-access-fcklq\" (UniqueName: \"kubernetes.io/projected/5e7cf3af-afa1-4308-b29e-d042c55df1e7-kube-api-access-fcklq\") pod \"5e7cf3af-afa1-4308-b29e-d042c55df1e7\" (UID: \"5e7cf3af-afa1-4308-b29e-d042c55df1e7\") "
Oct 31 13:22:26.447620 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:26.447593    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e7cf3af-afa1-4308-b29e-d042c55df1e7-kube-api-access-fcklq" pod "5e7cf3af-afa1-4308-b29e-d042c55df1e7" (UID: "5e7cf3af-afa1-4308-b29e-d042c55df1e7"). InnerVolumeSpecName "kube-api-access-fcklq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:22:26.546351 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:26.546323    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcklq\" (UniqueName: \"kubernetes.io/projected/5e7cf3af-afa1-4308-b29e-d042c55df1e7-kube-api-access-fcklq\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:22:26.753024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:26.752995    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03065bbf-072a-4d57-a919-c77a74478237" path="/var/lib/kubelet/pods/03065bbf-072a-4d57-a919-c77a74478237/volumes"
Oct 31 13:22:27.005136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:27.005057    4428 generic.go:356] "Generic (PLEG): container finished" podID="5e7cf3af-afa1-4308-b29e-d042c55df1e7" containerID="b084deebafabc2a56e50f14ac27a5a69b21de169a0e8532f1634b180831a0ac1" exitCode=0
Oct 31 13:22:27.005136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:27.005117    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-m4vph"
Oct 31 13:22:27.005595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:27.005150    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-m4vph" event={"ID":"5e7cf3af-afa1-4308-b29e-d042c55df1e7","Type":"ContainerDied","Data":"b084deebafabc2a56e50f14ac27a5a69b21de169a0e8532f1634b180831a0ac1"}
Oct 31 13:22:27.005595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:27.005190    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-m4vph" event={"ID":"5e7cf3af-afa1-4308-b29e-d042c55df1e7","Type":"ContainerDied","Data":"cd16aea33539fb6824502c4be46f6c11d6a9aa2cba6d3f4927c13af7fbd5ad9c"}
Oct 31 13:22:27.005595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:27.005211    4428 scope.go:122] "RemoveContainer" containerID="b084deebafabc2a56e50f14ac27a5a69b21de169a0e8532f1634b180831a0ac1"
Oct 31 13:22:27.016146 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:27.016126    4428 scope.go:122] "RemoveContainer" containerID="b084deebafabc2a56e50f14ac27a5a69b21de169a0e8532f1634b180831a0ac1"
Oct 31 13:22:27.016427 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:22:27.016403    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b084deebafabc2a56e50f14ac27a5a69b21de169a0e8532f1634b180831a0ac1\": container with ID starting with b084deebafabc2a56e50f14ac27a5a69b21de169a0e8532f1634b180831a0ac1 not found: ID does not exist" containerID="b084deebafabc2a56e50f14ac27a5a69b21de169a0e8532f1634b180831a0ac1"
Oct 31 13:22:27.016473 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:27.016441    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b084deebafabc2a56e50f14ac27a5a69b21de169a0e8532f1634b180831a0ac1"} err="failed to get container status \"b084deebafabc2a56e50f14ac27a5a69b21de169a0e8532f1634b180831a0ac1\": rpc error: code = NotFound desc = could not find container \"b084deebafabc2a56e50f14ac27a5a69b21de169a0e8532f1634b180831a0ac1\": container with ID starting with b084deebafabc2a56e50f14ac27a5a69b21de169a0e8532f1634b180831a0ac1 not found: ID does not exist"
Oct 31 13:22:27.025155 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:27.025132    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m4vph"]
Oct 31 13:22:27.029629 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:27.029608    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m4vph"]
Oct 31 13:22:28.753413 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:28.753380    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e7cf3af-afa1-4308-b29e-d042c55df1e7" path="/var/lib/kubelet/pods/5e7cf3af-afa1-4308-b29e-d042c55df1e7/volumes"
Oct 31 13:22:48.738145 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:22:48.738106    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:22:53.692442 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.692407    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"]
Oct 31 13:22:53.692895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.692718    4428 cpu_manager.go:414] "containerMap: removing container" podUID="5e7cf3af-afa1-4308-b29e-d042c55df1e7" containerName="authorino"
Oct 31 13:22:53.692895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.692729    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e7cf3af-afa1-4308-b29e-d042c55df1e7" containerName="authorino"
Oct 31 13:22:53.692895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.692809    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e7cf3af-afa1-4308-b29e-d042c55df1e7" containerName="authorino"
Oct 31 13:22:53.695707 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.695688    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.700990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.700969    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-r8f7r\""
Oct 31 13:22:53.714109 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.714086    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"]
Oct 31 13:22:53.717339 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.717316    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"]
Oct 31 13:22:53.717437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.717421    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.729106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.729081    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"]
Oct 31 13:22:53.732342 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.732316    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.732661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.732357    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.732661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.732599    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.732818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.732672    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.732818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.732711    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.732818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.732750    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.732818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.732772    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.732818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.732805    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlqrj\" (UniqueName: \"kubernetes.io/projected/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-kube-api-access-zlqrj\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.733079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.732851    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.833373 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.833336    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.833567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.833383    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca1c7578-b050-429c-8a3d-6d771125649c-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.833567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.833419    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.833567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.833435    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.833567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.833478    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.833567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.833548    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca1c7578-b050-429c-8a3d-6d771125649c-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.833845 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.833615    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl2tp\" (UniqueName: \"kubernetes.io/projected/ca1c7578-b050-429c-8a3d-6d771125649c-kube-api-access-cl2tp\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.833845 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.833657    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca1c7578-b050-429c-8a3d-6d771125649c-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.833845 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.833702    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca1c7578-b050-429c-8a3d-6d771125649c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.833845 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.833738    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.833845 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.833768    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.833845 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.833795    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.833845 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.833811    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca1c7578-b050-429c-8a3d-6d771125649c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.834235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.833905    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca1c7578-b050-429c-8a3d-6d771125649c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.834235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.834082    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca1c7578-b050-429c-8a3d-6d771125649c-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.834235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.834110    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.834235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.834175    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.834235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.834153    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.834400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.834342    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.834400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.834352    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.834400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.834371    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlqrj\" (UniqueName: \"kubernetes.io/projected/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-kube-api-access-zlqrj\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.834490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.834398    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca1c7578-b050-429c-8a3d-6d771125649c-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.834818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.834796    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.836054 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.836024    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.836265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.836248    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.846613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.846586    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.846913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.846892    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlqrj\" (UniqueName: \"kubernetes.io/projected/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-kube-api-access-zlqrj\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vcml7\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:53.935590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.935563    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca1c7578-b050-429c-8a3d-6d771125649c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.935679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.935594    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca1c7578-b050-429c-8a3d-6d771125649c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.935679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.935612    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca1c7578-b050-429c-8a3d-6d771125649c-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.935679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.935632    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca1c7578-b050-429c-8a3d-6d771125649c-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.935840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.935758    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca1c7578-b050-429c-8a3d-6d771125649c-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.935840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.935820    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca1c7578-b050-429c-8a3d-6d771125649c-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.935968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.935851    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl2tp\" (UniqueName: \"kubernetes.io/projected/ca1c7578-b050-429c-8a3d-6d771125649c-kube-api-access-cl2tp\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.935968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.935913    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca1c7578-b050-429c-8a3d-6d771125649c-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.935968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.935946    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca1c7578-b050-429c-8a3d-6d771125649c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.936120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.936079    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca1c7578-b050-429c-8a3d-6d771125649c-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.936425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.936401    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca1c7578-b050-429c-8a3d-6d771125649c-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.936524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.936429    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca1c7578-b050-429c-8a3d-6d771125649c-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.936524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.936425    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca1c7578-b050-429c-8a3d-6d771125649c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.936617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.936543    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca1c7578-b050-429c-8a3d-6d771125649c-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.937796 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.937775    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca1c7578-b050-429c-8a3d-6d771125649c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.938196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.938178    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca1c7578-b050-429c-8a3d-6d771125649c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.948912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.948847    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca1c7578-b050-429c-8a3d-6d771125649c-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:53.949167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:53.949150    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl2tp\" (UniqueName: \"kubernetes.io/projected/ca1c7578-b050-429c-8a3d-6d771125649c-kube-api-access-cl2tp\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-66bvk\" (UID: \"ca1c7578-b050-429c-8a3d-6d771125649c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:54.007997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:54.007965    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:54.032818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:54.032799    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:54.290796 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:54.290771    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"]
Oct 31 13:22:54.292632 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:22:54.292608    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ecf6cba_0a6c_40b4_a7f1_94e2e23fcfd0.slice/crio-7763b10f1b8e8261dd743887ca8555ae80a04816459f178e5ca539a7f3637416 WatchSource:0}: Error finding container 7763b10f1b8e8261dd743887ca8555ae80a04816459f178e5ca539a7f3637416: Status 404 returned error can't find the container with id 7763b10f1b8e8261dd743887ca8555ae80a04816459f178e5ca539a7f3637416
Oct 31 13:22:54.323098 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:54.323078    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"]
Oct 31 13:22:54.324917 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:22:54.324896    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1c7578_b050_429c_8a3d_6d771125649c.slice/crio-2b8946d75f81d7bc3cdd527a9ce38bbac1f875eb5ca857289a17284514ee6cec WatchSource:0}: Error finding container 2b8946d75f81d7bc3cdd527a9ce38bbac1f875eb5ca857289a17284514ee6cec: Status 404 returned error can't find the container with id 2b8946d75f81d7bc3cdd527a9ce38bbac1f875eb5ca857289a17284514ee6cec
Oct 31 13:22:55.120804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:55.120766    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk" event={"ID":"ca1c7578-b050-429c-8a3d-6d771125649c","Type":"ContainerStarted","Data":"2b8946d75f81d7bc3cdd527a9ce38bbac1f875eb5ca857289a17284514ee6cec"}
Oct 31 13:22:55.121855 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:55.121832    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7" event={"ID":"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0","Type":"ContainerStarted","Data":"7763b10f1b8e8261dd743887ca8555ae80a04816459f178e5ca539a7f3637416"}
Oct 31 13:22:57.451360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:57.451327    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:22:57.451634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:57.451397    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:22:57.451634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:57.451426    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:22:57.546095 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:57.546052    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:22:57.546219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:57.546149    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:22:57.546219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:57.546194    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:22:58.138434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:58.138398    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk" event={"ID":"ca1c7578-b050-429c-8a3d-6d771125649c","Type":"ContainerStarted","Data":"95709c40463c2f678044859b543f678d2e386254d2c2e73298e72730e07f52e0"}
Oct 31 13:22:58.139878 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:58.139840    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7" event={"ID":"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0","Type":"ContainerStarted","Data":"21dd50dc671e562caf98d2302522f322858d85c127b02fb507b6a36edaec3fc7"}
Oct 31 13:22:58.168891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:58.167824    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk" podStartSLOduration=2.044080882 podStartE2EDuration="5.167802757s" totalImagesPullingTime="3.123721875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:22:58.167242674 +0000 UTC m=+1175.565591298" watchObservedRunningTime="2026-10-31 13:22:58.167802757 +0000 UTC m=+1175.566151383"
Oct 31 13:22:58.212672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:58.211980    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7" podStartSLOduration=1.961559005 podStartE2EDuration="5.211962697s" totalImagesPullingTime="3.250403692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:22:58.211234205 +0000 UTC m=+1175.609582829" watchObservedRunningTime="2026-10-31 13:22:58.211962697 +0000 UTC m=+1175.610311325"
Oct 31 13:22:59.008308 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:59.008170    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:22:59.009614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:59.009587    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-vcml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Oct 31 13:22:59.009739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:59.009647    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7" podUID="0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Oct 31 13:22:59.033684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:59.033635    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:22:59.034496 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:59.034468    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-66bvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 31 13:22:59.034585 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:22:59.034513    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk" podUID="ca1c7578-b050-429c-8a3d-6d771125649c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 31 13:23:00.008747 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:00.008714    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-vcml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Oct 31 13:23:00.009154 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:00.008764    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7" podUID="0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Oct 31 13:23:00.033349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:00.033325    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-66bvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 31 13:23:00.033430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:00.033369    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk" podUID="ca1c7578-b050-429c-8a3d-6d771125649c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 31 13:23:00.832584 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:00.832545    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:23:01.001652 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:01.001618    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:23:01.008827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:01.008805    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-vcml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Oct 31 13:23:01.009161 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:01.008850    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7" podUID="0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Oct 31 13:23:01.033992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:01.033965    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-66bvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 31 13:23:01.034143 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:01.034030    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk" podUID="ca1c7578-b050-429c-8a3d-6d771125649c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 31 13:23:02.008371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:02.008329    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-vcml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Oct 31 13:23:02.008561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:02.008385    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7" podUID="0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Oct 31 13:23:02.033466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:02.033443    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-66bvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 31 13:23:02.033780 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:02.033486    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk" podUID="ca1c7578-b050-429c-8a3d-6d771125649c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 31 13:23:03.009206 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:03.009164    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-vcml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body=
Oct 31 13:23:03.009398 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:03.009234    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7" podUID="0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused"
Oct 31 13:23:03.033567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:03.033538    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-66bvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 31 13:23:03.034031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:03.033590    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk" podUID="ca1c7578-b050-429c-8a3d-6d771125649c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 31 13:23:04.012666 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:04.012628    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:23:04.013031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:04.013010    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:23:04.013699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:04.013681    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:23:04.037708 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:04.037483    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:23:04.038140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:04.038041    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:23:04.038621 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:04.038604    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk"
Oct 31 13:23:08.738790 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.738749    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"]
Oct 31 13:23:08.745647 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.745621    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.758893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.758831    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"]
Oct 31 13:23:08.787328 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.787292    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.787328 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.787321    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.787544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.787403    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.787544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.787435    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64mbl\" (UniqueName: \"kubernetes.io/projected/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-kube-api-access-64mbl\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.787544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.787468    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.787716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.787548    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.787716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.787584    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.787716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.787658    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.787716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.787698    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.888738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.888707    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.888738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.888744    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.889001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.888764    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.889001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.888805    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.889001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.888845    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.889001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.888895    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.889224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.889008    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.889224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.889069    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.889224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.889110    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64mbl\" (UniqueName: \"kubernetes.io/projected/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-kube-api-access-64mbl\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.889392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.889310    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.889392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.889334    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.889505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.889453    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.889566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.889545    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.889751 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.889723    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.891495 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.891477    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.891764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.891746    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.901567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.901547    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:08.901775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:08.901759    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64mbl\" (UniqueName: \"kubernetes.io/projected/aaa34a48-48ac-4d05-8d9d-0d0c0a870f88-kube-api-access-64mbl\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-n56bm\" (UID: \"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:09.064656 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:09.064599    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:09.346673 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:09.346625    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"]
Oct 31 13:23:09.349004 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:23:09.348979    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa34a48_48ac_4d05_8d9d_0d0c0a870f88.slice/crio-e937cd068d703c0118780728439266abe7d23b7850e93f3cd49e3aba7e2e0be2 WatchSource:0}: Error finding container e937cd068d703c0118780728439266abe7d23b7850e93f3cd49e3aba7e2e0be2: Status 404 returned error can't find the container with id e937cd068d703c0118780728439266abe7d23b7850e93f3cd49e3aba7e2e0be2
Oct 31 13:23:09.351959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:09.351928    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:23:09.352029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:09.351992    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:23:09.352029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:09.352022    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:23:10.238351 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:10.238310    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm" event={"ID":"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88","Type":"ContainerStarted","Data":"f76ef2dcecd67677dcf1c9e2c7fcc7a949e3e993232b8cdb589bb89f5782c4a4"}
Oct 31 13:23:10.238805 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:10.238397    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm" event={"ID":"aaa34a48-48ac-4d05-8d9d-0d0c0a870f88","Type":"ContainerStarted","Data":"e937cd068d703c0118780728439266abe7d23b7850e93f3cd49e3aba7e2e0be2"}
Oct 31 13:23:10.272142 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:10.272090    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm" podStartSLOduration=2.272077011 podStartE2EDuration="2.272077011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:10.270326301 +0000 UTC m=+1187.668674923" watchObservedRunningTime="2026-10-31 13:23:10.272077011 +0000 UTC m=+1187.670425635"
Oct 31 13:23:11.068893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:11.068224    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:11.070987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:11.070960    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-n56bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 31 13:23:11.071133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:11.071025    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm" podUID="aaa34a48-48ac-4d05-8d9d-0d0c0a870f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 31 13:23:12.062276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:12.062224    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:23:12.069243 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:12.069216    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-n56bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 31 13:23:12.069432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:12.069406    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm" podUID="aaa34a48-48ac-4d05-8d9d-0d0c0a870f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 31 13:23:13.065891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:13.065232    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-n56bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 31 13:23:13.065891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:13.065295    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm" podUID="aaa34a48-48ac-4d05-8d9d-0d0c0a870f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 31 13:23:14.065313 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:14.065274    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-n56bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 31 13:23:14.065605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:14.065574    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm" podUID="aaa34a48-48ac-4d05-8d9d-0d0c0a870f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 31 13:23:15.068894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:15.067957    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-n56bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 31 13:23:15.068894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:15.068023    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm" podUID="aaa34a48-48ac-4d05-8d9d-0d0c0a870f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 31 13:23:16.069172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:16.069142    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:16.069693 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:16.069455    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:16.070296 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:16.070270    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm"
Oct 31 13:23:17.306130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.306096    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"]
Oct 31 13:23:17.314265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.314242    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.325176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.325157    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"]
Oct 31 13:23:17.326856 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.326836    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 13:23:17.327144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.327122    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 13:23:17.327144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.327141    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9845x\""
Oct 31 13:23:17.327275 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.327262    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 31 13:23:17.361185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.361158    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea978050-20b0-4148-ab41-75844fe1b637-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.361312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.361229    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea978050-20b0-4148-ab41-75844fe1b637-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.361380 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.361329    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea978050-20b0-4148-ab41-75844fe1b637-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.361380 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.361364    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v4qg\" (UniqueName: \"kubernetes.io/projected/ea978050-20b0-4148-ab41-75844fe1b637-kube-api-access-9v4qg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.361476 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.361454    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea978050-20b0-4148-ab41-75844fe1b637-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.361576 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.361556    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea978050-20b0-4148-ab41-75844fe1b637-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.398838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.398816    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:23:17.462984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.462954    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea978050-20b0-4148-ab41-75844fe1b637-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.463160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.463002    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea978050-20b0-4148-ab41-75844fe1b637-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.463160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.463035    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea978050-20b0-4148-ab41-75844fe1b637-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.463160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.463051    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea978050-20b0-4148-ab41-75844fe1b637-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.463160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.463066    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v4qg\" (UniqueName: \"kubernetes.io/projected/ea978050-20b0-4148-ab41-75844fe1b637-kube-api-access-9v4qg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.463160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.463097    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea978050-20b0-4148-ab41-75844fe1b637-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.463497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.463356    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea978050-20b0-4148-ab41-75844fe1b637-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.463497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.463473    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea978050-20b0-4148-ab41-75844fe1b637-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.463584 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.463510    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea978050-20b0-4148-ab41-75844fe1b637-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.465428 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.465407    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea978050-20b0-4148-ab41-75844fe1b637-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.465600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.465582    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea978050-20b0-4148-ab41-75844fe1b637-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.476535 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.476514    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v4qg\" (UniqueName: \"kubernetes.io/projected/ea978050-20b0-4148-ab41-75844fe1b637-kube-api-access-9v4qg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hql47\" (UID: \"ea978050-20b0-4148-ab41-75844fe1b637\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.627974 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.627893    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:17.909008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:17.908931    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"]
Oct 31 13:23:17.913026 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:23:17.912990    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea978050_20b0_4148_ab41_75844fe1b637.slice/crio-4fc106aa95480ce77bd020e54bb4a2b9ef2082ca7f6b7efbda739fd45f613d9d WatchSource:0}: Error finding container 4fc106aa95480ce77bd020e54bb4a2b9ef2082ca7f6b7efbda739fd45f613d9d: Status 404 returned error can't find the container with id 4fc106aa95480ce77bd020e54bb4a2b9ef2082ca7f6b7efbda739fd45f613d9d
Oct 31 13:23:18.330893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:18.320474    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47" event={"ID":"ea978050-20b0-4148-ab41-75844fe1b637","Type":"ContainerStarted","Data":"4fc106aa95480ce77bd020e54bb4a2b9ef2082ca7f6b7efbda739fd45f613d9d"}
Oct 31 13:23:18.749124 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:23:18.749090    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:23:19.151711 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.151642    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"]
Oct 31 13:23:19.155144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.155123    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.164450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.164430    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 31 13:23:19.169675 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.169646    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"]
Oct 31 13:23:19.259775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.259734    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:23:19.300216 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.300185    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0ac048-d186-4f3c-929d-eb870afbc801-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.300405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.300249    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0ac048-d186-4f3c-929d-eb870afbc801-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.300405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.300360    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0ac048-d186-4f3c-929d-eb870afbc801-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.300488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.300418    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0ac048-d186-4f3c-929d-eb870afbc801-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.300525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.300484    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbgrr\" (UniqueName: \"kubernetes.io/projected/cd0ac048-d186-4f3c-929d-eb870afbc801-kube-api-access-vbgrr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.300614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.300534    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0ac048-d186-4f3c-929d-eb870afbc801-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.401653 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.401615    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0ac048-d186-4f3c-929d-eb870afbc801-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.402193 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.401672    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0ac048-d186-4f3c-929d-eb870afbc801-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.402193 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.401710    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0ac048-d186-4f3c-929d-eb870afbc801-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.402193 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.401751    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0ac048-d186-4f3c-929d-eb870afbc801-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.402193 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.401811    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbgrr\" (UniqueName: \"kubernetes.io/projected/cd0ac048-d186-4f3c-929d-eb870afbc801-kube-api-access-vbgrr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.402193 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.401853    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0ac048-d186-4f3c-929d-eb870afbc801-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.402193 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.402154    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0ac048-d186-4f3c-929d-eb870afbc801-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.402490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.402251    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0ac048-d186-4f3c-929d-eb870afbc801-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.402490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.402265    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0ac048-d186-4f3c-929d-eb870afbc801-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.405247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.405224    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0ac048-d186-4f3c-929d-eb870afbc801-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.405375 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.405257    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0ac048-d186-4f3c-929d-eb870afbc801-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.417132 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.417108    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbgrr\" (UniqueName: \"kubernetes.io/projected/cd0ac048-d186-4f3c-929d-eb870afbc801-kube-api-access-vbgrr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59\" (UID: \"cd0ac048-d186-4f3c-929d-eb870afbc801\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.473748 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.473712    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:19.770935 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:19.770903    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"]
Oct 31 13:23:19.782545 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:23:19.782249    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0ac048_d186_4f3c_929d_eb870afbc801.slice/crio-3563610e02958c975003b11f815d88d2f20207a0480e6c9904afc61d7ea899f8 WatchSource:0}: Error finding container 3563610e02958c975003b11f815d88d2f20207a0480e6c9904afc61d7ea899f8: Status 404 returned error can't find the container with id 3563610e02958c975003b11f815d88d2f20207a0480e6c9904afc61d7ea899f8
Oct 31 13:23:20.333322 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:20.333278    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47" event={"ID":"ea978050-20b0-4148-ab41-75844fe1b637","Type":"ContainerStarted","Data":"b4528c28c900cbb2738d40f6089ef788c2aa9c3b5189dc23913f3a6efe321339"}
Oct 31 13:23:20.334592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:20.334572    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59" event={"ID":"cd0ac048-d186-4f3c-929d-eb870afbc801","Type":"ContainerStarted","Data":"3563610e02958c975003b11f815d88d2f20207a0480e6c9904afc61d7ea899f8"}
Oct 31 13:23:20.359889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:20.357606    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47" podStartSLOduration=1.5510617230000001 podStartE2EDuration="3.357586983s" totalImagesPullingTime="1.80652526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:20.355600848 +0000 UTC m=+1197.753949467" watchObservedRunningTime="2026-10-31 13:23:20.357586983 +0000 UTC m=+1197.755935610"
Oct 31 13:23:20.628217 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:20.628175    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:21.340072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:21.340030    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59" event={"ID":"cd0ac048-d186-4f3c-929d-eb870afbc801","Type":"ContainerStarted","Data":"c867ef145d195b2fa960cee13751ee94edda6e56d64832e238f5c635a561606b"}
Oct 31 13:23:21.360938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:21.360885    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59" podStartSLOduration=1.946650177 podStartE2EDuration="2.360848839s" totalImagesPullingTime="414.198662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:21.360775205 +0000 UTC m=+1198.759123829" watchObservedRunningTime="2026-10-31 13:23:21.360848839 +0000 UTC m=+1198.759197464"
Oct 31 13:23:21.475274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:21.475007    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:23.839997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:23.839966    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-8b7s6"]
Oct 31 13:23:23.844929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:23.844900    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:23.849403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:23.849380    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 31 13:23:23.854407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:23.854386    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-8b7s6"]
Oct 31 13:23:23.945536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:23.945506    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:23:23.953699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:23.953666    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/149a9b52-ef95-4569-b947-e6e14d707339-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:23.953790 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:23.953708    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/149a9b52-ef95-4569-b947-e6e14d707339-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:23.953790 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:23.953733    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/149a9b52-ef95-4569-b947-e6e14d707339-home\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:23.953860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:23.953825    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/149a9b52-ef95-4569-b947-e6e14d707339-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:23.953919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:23.953863    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv8pr\" (UniqueName: \"kubernetes.io/projected/149a9b52-ef95-4569-b947-e6e14d707339-kube-api-access-tv8pr\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:23.953971 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:23.953957    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/149a9b52-ef95-4569-b947-e6e14d707339-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:24.054563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:24.054530    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/149a9b52-ef95-4569-b947-e6e14d707339-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:24.054744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:24.054571    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/149a9b52-ef95-4569-b947-e6e14d707339-home\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:24.054744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:24.054598    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/149a9b52-ef95-4569-b947-e6e14d707339-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:24.054744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:24.054626    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv8pr\" (UniqueName: \"kubernetes.io/projected/149a9b52-ef95-4569-b947-e6e14d707339-kube-api-access-tv8pr\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:24.054744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:24.054656    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/149a9b52-ef95-4569-b947-e6e14d707339-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:24.054744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:24.054697    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/149a9b52-ef95-4569-b947-e6e14d707339-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:24.055070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:24.055047    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/149a9b52-ef95-4569-b947-e6e14d707339-home\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:24.055133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:24.055095    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/149a9b52-ef95-4569-b947-e6e14d707339-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:24.055133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:24.055118    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/149a9b52-ef95-4569-b947-e6e14d707339-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:24.057153 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:24.057127    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/149a9b52-ef95-4569-b947-e6e14d707339-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:24.057323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:24.057308    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/149a9b52-ef95-4569-b947-e6e14d707339-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:24.068210 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:24.068191    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv8pr\" (UniqueName: \"kubernetes.io/projected/149a9b52-ef95-4569-b947-e6e14d707339-kube-api-access-tv8pr\") pod \"echo-simulator-kserve-64b6f8f96c-8b7s6\" (UID: \"149a9b52-ef95-4569-b947-e6e14d707339\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:24.158673 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:24.158614    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:24.438547 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:24.438521    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-8b7s6"]
Oct 31 13:23:24.440762 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:23:24.440717    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149a9b52_ef95_4569_b947_e6e14d707339.slice/crio-27f56331ebbcfca7b8c18eadf833ba67df2560d0fca6ce2ad7d331378e3735c1 WatchSource:0}: Error finding container 27f56331ebbcfca7b8c18eadf833ba67df2560d0fca6ce2ad7d331378e3735c1: Status 404 returned error can't find the container with id 27f56331ebbcfca7b8c18eadf833ba67df2560d0fca6ce2ad7d331378e3735c1
Oct 31 13:23:24.516582 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:24.516545    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:23:25.360354 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:25.360315    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6" event={"ID":"149a9b52-ef95-4569-b947-e6e14d707339","Type":"ContainerStarted","Data":"27f56331ebbcfca7b8c18eadf833ba67df2560d0fca6ce2ad7d331378e3735c1"}
Oct 31 13:23:26.485629 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:26.485598    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:23:27.388957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:27.384350    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6" event={"ID":"149a9b52-ef95-4569-b947-e6e14d707339","Type":"ContainerStarted","Data":"c9e71cb53a23f75bead22341dae8ba84129f21f3b17624ca6897bbb32ccb8034"}
Oct 31 13:23:27.414262 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:27.414205    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6" podStartSLOduration=1.798373152 podStartE2EDuration="4.414186503s" totalImagesPullingTime="2.615813351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:27.413998244 +0000 UTC m=+1204.812346893" watchObservedRunningTime="2026-10-31 13:23:27.414186503 +0000 UTC m=+1204.812535128"
Oct 31 13:23:27.628810 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:27.628777    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:27.641187 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:27.641119    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:28.159142 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:28.159105    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:28.402791 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:28.402761    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47"
Oct 31 13:23:28.980358 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:28.980319    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"]
Oct 31 13:23:28.984831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:28.984803    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:28.999583 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:28.999559    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"]
Oct 31 13:23:29.005877 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:28.999990    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 31 13:23:29.094967 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.094924    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:23:29.111560 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.111529    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c86b906-a276-4e67-8e58-79a17118f5fd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.111736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.111572    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c86b906-a276-4e67-8e58-79a17118f5fd-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.111736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.111607    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c86b906-a276-4e67-8e58-79a17118f5fd-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.111736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.111654    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c86b906-a276-4e67-8e58-79a17118f5fd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.111736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.111710    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c86b906-a276-4e67-8e58-79a17118f5fd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.112024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.111754    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lt6l\" (UniqueName: \"kubernetes.io/projected/5c86b906-a276-4e67-8e58-79a17118f5fd-kube-api-access-6lt6l\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.212538 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.212498    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lt6l\" (UniqueName: \"kubernetes.io/projected/5c86b906-a276-4e67-8e58-79a17118f5fd-kube-api-access-6lt6l\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.212756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.212620    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c86b906-a276-4e67-8e58-79a17118f5fd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.212756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.212654    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c86b906-a276-4e67-8e58-79a17118f5fd-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.212756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.212685    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c86b906-a276-4e67-8e58-79a17118f5fd-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.212756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.212744    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c86b906-a276-4e67-8e58-79a17118f5fd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.213040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.212811    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c86b906-a276-4e67-8e58-79a17118f5fd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.213178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.213147    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c86b906-a276-4e67-8e58-79a17118f5fd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.213661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.213624    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c86b906-a276-4e67-8e58-79a17118f5fd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.213998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.213976    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c86b906-a276-4e67-8e58-79a17118f5fd-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.216295 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.216250    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c86b906-a276-4e67-8e58-79a17118f5fd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.216393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.216326    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c86b906-a276-4e67-8e58-79a17118f5fd-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.226204 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.226174    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lt6l\" (UniqueName: \"kubernetes.io/projected/5c86b906-a276-4e67-8e58-79a17118f5fd-kube-api-access-6lt6l\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg\" (UID: \"5c86b906-a276-4e67-8e58-79a17118f5fd\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.309104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.309025    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:29.509920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.488283    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:29.529495 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.529244    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:29.723889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:29.717613    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"]
Oct 31 13:23:29.731033 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:23:29.731001    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c86b906_a276_4e67_8e58_79a17118f5fd.slice/crio-9228108082a70f973223344568506eb98ab68961a89a555cedc71bededd59be7 WatchSource:0}: Error finding container 9228108082a70f973223344568506eb98ab68961a89a555cedc71bededd59be7: Status 404 returned error can't find the container with id 9228108082a70f973223344568506eb98ab68961a89a555cedc71bededd59be7
Oct 31 13:23:30.528404 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.528364    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg" event={"ID":"5c86b906-a276-4e67-8e58-79a17118f5fd","Type":"ContainerStarted","Data":"058042c48cd2850fc8058160a9541da848d40741c392a2d804f6991c199904e1"}
Oct 31 13:23:30.529692 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.529654    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg" event={"ID":"5c86b906-a276-4e67-8e58-79a17118f5fd","Type":"ContainerStarted","Data":"9228108082a70f973223344568506eb98ab68961a89a555cedc71bededd59be7"}
Oct 31 13:23:30.533077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.533055    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:23:30.556970 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.556781    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg" podStartSLOduration=2.16526824 podStartE2EDuration="2.556762211s" totalImagesPullingTime="391.493971ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:30.554081755 +0000 UTC m=+1207.952430379" watchObservedRunningTime="2026-10-31 13:23:30.556762211 +0000 UTC m=+1207.955110836"
Oct 31 13:23:30.642842 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.641270    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"]
Oct 31 13:23:30.644911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.644885    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.652143 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.652116    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 31 13:23:30.660320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.660292    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"]
Oct 31 13:23:30.744325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.744292    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c20e3218-9e4a-43a1-b46e-050de8eb25c4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.744509 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.744418    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c20e3218-9e4a-43a1-b46e-050de8eb25c4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.744509 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.744463    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c20e3218-9e4a-43a1-b46e-050de8eb25c4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.744509 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.744481    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcvgx\" (UniqueName: \"kubernetes.io/projected/c20e3218-9e4a-43a1-b46e-050de8eb25c4-kube-api-access-bcvgx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.744509 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.744502    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c20e3218-9e4a-43a1-b46e-050de8eb25c4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.744749 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.744569    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c20e3218-9e4a-43a1-b46e-050de8eb25c4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.773597 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.773569    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:23:30.845407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.845309    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c20e3218-9e4a-43a1-b46e-050de8eb25c4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.845407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.845373    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c20e3218-9e4a-43a1-b46e-050de8eb25c4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.845407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.845401    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcvgx\" (UniqueName: \"kubernetes.io/projected/c20e3218-9e4a-43a1-b46e-050de8eb25c4-kube-api-access-bcvgx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.845683 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.845429    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c20e3218-9e4a-43a1-b46e-050de8eb25c4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.845683 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.845479    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c20e3218-9e4a-43a1-b46e-050de8eb25c4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.845683 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.845516    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c20e3218-9e4a-43a1-b46e-050de8eb25c4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.846274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.846252    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c20e3218-9e4a-43a1-b46e-050de8eb25c4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.846429 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.846304    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c20e3218-9e4a-43a1-b46e-050de8eb25c4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.846712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.846689    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c20e3218-9e4a-43a1-b46e-050de8eb25c4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.849249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.849226    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c20e3218-9e4a-43a1-b46e-050de8eb25c4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.849970 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.849952    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c20e3218-9e4a-43a1-b46e-050de8eb25c4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.861205 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.861178    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcvgx\" (UniqueName: \"kubernetes.io/projected/c20e3218-9e4a-43a1-b46e-050de8eb25c4-kube-api-access-bcvgx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-596gr\" (UID: \"c20e3218-9e4a-43a1-b46e-050de8eb25c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:30.969033 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:30.968993    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:31.271321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:31.271288    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"]
Oct 31 13:23:31.275059 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:23:31.275020    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20e3218_9e4a_43a1_b46e_050de8eb25c4.slice/crio-483d3c564948dc4a061c2abe8e5467ea561cd70c03151bb2117ecbd6bfef6311 WatchSource:0}: Error finding container 483d3c564948dc4a061c2abe8e5467ea561cd70c03151bb2117ecbd6bfef6311: Status 404 returned error can't find the container with id 483d3c564948dc4a061c2abe8e5467ea561cd70c03151bb2117ecbd6bfef6311
Oct 31 13:23:31.309702 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:31.309664    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:31.533266 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:31.533171    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr" event={"ID":"c20e3218-9e4a-43a1-b46e-050de8eb25c4","Type":"ContainerStarted","Data":"483d3c564948dc4a061c2abe8e5467ea561cd70c03151bb2117ecbd6bfef6311"}
Oct 31 13:23:32.538927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:32.538886    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr" event={"ID":"c20e3218-9e4a-43a1-b46e-050de8eb25c4","Type":"ContainerStarted","Data":"7ffbe7bf39ed3f9c6419d19848af16245c6097311df75762abf7fa391327e874"}
Oct 31 13:23:32.969348 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:32.969314    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:33.833989 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:33.833955    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:23:34.159714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:34.159599    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:34.179227 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:34.179200    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:34.201531 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:34.201463    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr" podStartSLOduration=3.926456616 podStartE2EDuration="4.201443674s" totalImagesPullingTime="274.987058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:32.566512232 +0000 UTC m=+1209.964860857" watchObservedRunningTime="2026-10-31 13:23:34.201443674 +0000 UTC m=+1211.599792300"
Oct 31 13:23:34.618892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:34.618175    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6"
Oct 31 13:23:39.309374 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:39.309339    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:39.321913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:39.321890    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:39.635659 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:39.635587    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg"
Oct 31 13:23:40.969510 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:40.969481    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:40.981233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:40.981213    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:41.645831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:41.645807    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:23:46.461069 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:46.461032    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-575b5b59b7-6gpnj"]
Oct 31 13:23:46.461606 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:46.461217    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-575b5b59b7-6gpnj" podUID="055782a8-f2d0-49a3-b475-ee6e8f09fcd3" containerName="authorino" containerID="cri-o://61d967d8d70a3b15833c44cde421b00f1158505566579347f737bb1c339aa78d" gracePeriod=30
Oct 31 13:23:46.663376 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:46.663346    4428 generic.go:356] "Generic (PLEG): container finished" podID="055782a8-f2d0-49a3-b475-ee6e8f09fcd3" containerID="61d967d8d70a3b15833c44cde421b00f1158505566579347f737bb1c339aa78d" exitCode=0
Oct 31 13:23:46.663516 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:46.663385    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-575b5b59b7-6gpnj" event={"ID":"055782a8-f2d0-49a3-b475-ee6e8f09fcd3","Type":"ContainerDied","Data":"61d967d8d70a3b15833c44cde421b00f1158505566579347f737bb1c339aa78d"}
Oct 31 13:23:46.710908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:46.710883    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-575b5b59b7-6gpnj"
Oct 31 13:23:46.785362 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:46.785297    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/055782a8-f2d0-49a3-b475-ee6e8f09fcd3-kube-api-access-4jcrc\" (UniqueName: \"kubernetes.io/projected/055782a8-f2d0-49a3-b475-ee6e8f09fcd3-kube-api-access-4jcrc\") pod \"055782a8-f2d0-49a3-b475-ee6e8f09fcd3\" (UID: \"055782a8-f2d0-49a3-b475-ee6e8f09fcd3\") "
Oct 31 13:23:46.785489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:46.785367    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/055782a8-f2d0-49a3-b475-ee6e8f09fcd3-tls-cert\" (UniqueName: \"kubernetes.io/secret/055782a8-f2d0-49a3-b475-ee6e8f09fcd3-tls-cert\") pod \"055782a8-f2d0-49a3-b475-ee6e8f09fcd3\" (UID: \"055782a8-f2d0-49a3-b475-ee6e8f09fcd3\") "
Oct 31 13:23:46.787427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:46.787403    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055782a8-f2d0-49a3-b475-ee6e8f09fcd3-kube-api-access-4jcrc" pod "055782a8-f2d0-49a3-b475-ee6e8f09fcd3" (UID: "055782a8-f2d0-49a3-b475-ee6e8f09fcd3"). InnerVolumeSpecName "kube-api-access-4jcrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:23:46.797222 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:46.797185    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/055782a8-f2d0-49a3-b475-ee6e8f09fcd3-tls-cert" pod "055782a8-f2d0-49a3-b475-ee6e8f09fcd3" (UID: "055782a8-f2d0-49a3-b475-ee6e8f09fcd3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:23:46.886057 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:46.886028    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jcrc\" (UniqueName: \"kubernetes.io/projected/055782a8-f2d0-49a3-b475-ee6e8f09fcd3-kube-api-access-4jcrc\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:23:46.886057 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:46.886054    4428 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/055782a8-f2d0-49a3-b475-ee6e8f09fcd3-tls-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:23:47.038706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.038617    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"]
Oct 31 13:23:47.039215 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.039199    4428 cpu_manager.go:414] "containerMap: removing container" podUID="055782a8-f2d0-49a3-b475-ee6e8f09fcd3" containerName="authorino"
Oct 31 13:23:47.039292 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.039219    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="055782a8-f2d0-49a3-b475-ee6e8f09fcd3" containerName="authorino"
Oct 31 13:23:47.039334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.039314    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="055782a8-f2d0-49a3-b475-ee6e8f09fcd3" containerName="authorino"
Oct 31 13:23:47.046670 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.044274    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"
Oct 31 13:23:47.053917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.053890    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 13:23:47.054953 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.054931    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"]
Oct 31 13:23:47.055126 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.055109    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-6rzt8\""
Oct 31 13:23:47.058295 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.058273    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 13:23:47.086839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.086817    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:23:47.087970 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.087949    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnrs9\" (UniqueName: \"kubernetes.io/projected/10f71b0f-cc6f-4d6a-add0-56ce2113fb47-kube-api-access-pnrs9\") pod \"ai-gateway-controller-66dd76bf94-qtxk2\" (UID: \"10f71b0f-cc6f-4d6a-add0-56ce2113fb47\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"
Oct 31 13:23:47.088078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.088062    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10f71b0f-cc6f-4d6a-add0-56ce2113fb47-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-qtxk2\" (UID: \"10f71b0f-cc6f-4d6a-add0-56ce2113fb47\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"
Oct 31 13:23:47.189206 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.189180    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10f71b0f-cc6f-4d6a-add0-56ce2113fb47-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-qtxk2\" (UID: \"10f71b0f-cc6f-4d6a-add0-56ce2113fb47\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"
Oct 31 13:23:47.189307 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.189243    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnrs9\" (UniqueName: \"kubernetes.io/projected/10f71b0f-cc6f-4d6a-add0-56ce2113fb47-kube-api-access-pnrs9\") pod \"ai-gateway-controller-66dd76bf94-qtxk2\" (UID: \"10f71b0f-cc6f-4d6a-add0-56ce2113fb47\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"
Oct 31 13:23:47.189347 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:23:47.189322    4428 secret.go:189] Couldn't get secret opendatahub/ai-gateway-controller-metrics-tls: secret "ai-gateway-controller-metrics-tls" not found
Oct 31 13:23:47.189399 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:23:47.189389    4428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10f71b0f-cc6f-4d6a-add0-56ce2113fb47-metrics-certs podName:10f71b0f-cc6f-4d6a-add0-56ce2113fb47 nodeName:}" failed. No retries permitted until 2026-10-31 13:23:47.689375228 +0000 UTC m=+1225.087723849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/10f71b0f-cc6f-4d6a-add0-56ce2113fb47-metrics-certs") pod "ai-gateway-controller-66dd76bf94-qtxk2" (UID: "10f71b0f-cc6f-4d6a-add0-56ce2113fb47") : secret "ai-gateway-controller-metrics-tls" not found
Oct 31 13:23:47.205256 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.205233    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnrs9\" (UniqueName: \"kubernetes.io/projected/10f71b0f-cc6f-4d6a-add0-56ce2113fb47-kube-api-access-pnrs9\") pod \"ai-gateway-controller-66dd76bf94-qtxk2\" (UID: \"10f71b0f-cc6f-4d6a-add0-56ce2113fb47\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"
Oct 31 13:23:47.394738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.394653    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:23:47.668602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.668529    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-575b5b59b7-6gpnj"
Oct 31 13:23:47.669019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.668533    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-575b5b59b7-6gpnj" event={"ID":"055782a8-f2d0-49a3-b475-ee6e8f09fcd3","Type":"ContainerDied","Data":"e707fd924205611593a77bea9678fcfb430b3b02e68fb91d4c070838e16ce0e4"}
Oct 31 13:23:47.669019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.668653    4428 scope.go:122] "RemoveContainer" containerID="61d967d8d70a3b15833c44cde421b00f1158505566579347f737bb1c339aa78d"
Oct 31 13:23:47.691030 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.691006    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-575b5b59b7-6gpnj"]
Oct 31 13:23:47.693548 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.693530    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10f71b0f-cc6f-4d6a-add0-56ce2113fb47-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-qtxk2\" (UID: \"10f71b0f-cc6f-4d6a-add0-56ce2113fb47\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"
Oct 31 13:23:47.696107 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.696080    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10f71b0f-cc6f-4d6a-add0-56ce2113fb47-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-qtxk2\" (UID: \"10f71b0f-cc6f-4d6a-add0-56ce2113fb47\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"
Oct 31 13:23:47.700078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.700057    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-575b5b59b7-6gpnj"]
Oct 31 13:23:47.961298 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:47.961277    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"
Oct 31 13:23:48.237087 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:48.237066    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"]
Oct 31 13:23:48.239199 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:23:48.239166    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f71b0f_cc6f_4d6a_add0_56ce2113fb47.slice/crio-c21ca4d8626ecb447b4d33732657648de5ad2038ccdb3092a34fd17fdfcbd6ae WatchSource:0}: Error finding container c21ca4d8626ecb447b4d33732657648de5ad2038ccdb3092a34fd17fdfcbd6ae: Status 404 returned error can't find the container with id c21ca4d8626ecb447b4d33732657648de5ad2038ccdb3092a34fd17fdfcbd6ae
Oct 31 13:23:48.675723 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:48.675642    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2" event={"ID":"10f71b0f-cc6f-4d6a-add0-56ce2113fb47","Type":"ContainerStarted","Data":"c21ca4d8626ecb447b4d33732657648de5ad2038ccdb3092a34fd17fdfcbd6ae"}
Oct 31 13:23:48.738590 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:23:48.738554    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:23:48.753688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:48.753662    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="055782a8-f2d0-49a3-b475-ee6e8f09fcd3" path="/var/lib/kubelet/pods/055782a8-f2d0-49a3-b475-ee6e8f09fcd3/volumes"
Oct 31 13:23:51.694221 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:51.694186    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2" event={"ID":"10f71b0f-cc6f-4d6a-add0-56ce2113fb47","Type":"ContainerStarted","Data":"41af66cf18cf550907843a6b03391f3a9aadf4e410deb586643f63be00144622"}
Oct 31 13:23:51.694806 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:51.694333    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"
Oct 31 13:23:51.714041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:51.713985    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2" podStartSLOduration=2.118677685 podStartE2EDuration="4.713969598s" totalImagesPullingTime="2.595291913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:51.712994791 +0000 UTC m=+1229.111343416" watchObservedRunningTime="2026-10-31 13:23:51.713969598 +0000 UTC m=+1229.112318222"
Oct 31 13:23:52.658275 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.658237    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:23:52.665559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.665532    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-km7m9"]
Oct 31 13:23:52.669093 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.669076    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-km7m9"
Oct 31 13:23:52.678512 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.678493    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-fl2m5\""
Oct 31 13:23:52.681227 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.681207    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-km7m9"]
Oct 31 13:23:52.697598 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.697569    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-w9wjk"]
Oct 31 13:23:52.701185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.701169    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-w9wjk"
Oct 31 13:23:52.709103 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.709079    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-w9wjk"]
Oct 31 13:23:52.738986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.738956    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 13:23:52.750066 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.750025    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e328741d-31fd-40db-8cbb-8c9b518f4cde-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-w9wjk\" (UID: \"e328741d-31fd-40db-8cbb-8c9b518f4cde\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w9wjk"
Oct 31 13:23:52.750319 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.750128    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m2tb\" (UniqueName: \"kubernetes.io/projected/e328741d-31fd-40db-8cbb-8c9b518f4cde-kube-api-access-5m2tb\") pod \"payload-processing-55fbbbf9dc-w9wjk\" (UID: \"e328741d-31fd-40db-8cbb-8c9b518f4cde\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w9wjk"
Oct 31 13:23:52.750319 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.750199    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db129029-9467-4431-bc52-40c0524ff214-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-km7m9\" (UID: \"db129029-9467-4431-bc52-40c0524ff214\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-km7m9"
Oct 31 13:23:52.750319 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.750300    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g46q\" (UniqueName: \"kubernetes.io/projected/db129029-9467-4431-bc52-40c0524ff214-kube-api-access-8g46q\") pod \"payload-pre-processing-7b98bb678c-km7m9\" (UID: \"db129029-9467-4431-bc52-40c0524ff214\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-km7m9"
Oct 31 13:23:52.813302 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.813266    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:23:52.851690 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.851659    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db129029-9467-4431-bc52-40c0524ff214-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-km7m9\" (UID: \"db129029-9467-4431-bc52-40c0524ff214\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-km7m9"
Oct 31 13:23:52.851880 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.851752    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g46q\" (UniqueName: \"kubernetes.io/projected/db129029-9467-4431-bc52-40c0524ff214-kube-api-access-8g46q\") pod \"payload-pre-processing-7b98bb678c-km7m9\" (UID: \"db129029-9467-4431-bc52-40c0524ff214\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-km7m9"
Oct 31 13:23:52.851880 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.851783    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e328741d-31fd-40db-8cbb-8c9b518f4cde-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-w9wjk\" (UID: \"e328741d-31fd-40db-8cbb-8c9b518f4cde\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w9wjk"
Oct 31 13:23:52.851880 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.851847    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m2tb\" (UniqueName: \"kubernetes.io/projected/e328741d-31fd-40db-8cbb-8c9b518f4cde-kube-api-access-5m2tb\") pod \"payload-processing-55fbbbf9dc-w9wjk\" (UID: \"e328741d-31fd-40db-8cbb-8c9b518f4cde\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w9wjk"
Oct 31 13:23:52.852489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.852472    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db129029-9467-4431-bc52-40c0524ff214-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-km7m9\" (UID: \"db129029-9467-4431-bc52-40c0524ff214\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-km7m9"
Oct 31 13:23:52.852529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.852494    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e328741d-31fd-40db-8cbb-8c9b518f4cde-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-w9wjk\" (UID: \"e328741d-31fd-40db-8cbb-8c9b518f4cde\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w9wjk"
Oct 31 13:23:52.865433 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.865400    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g46q\" (UniqueName: \"kubernetes.io/projected/db129029-9467-4431-bc52-40c0524ff214-kube-api-access-8g46q\") pod \"payload-pre-processing-7b98bb678c-km7m9\" (UID: \"db129029-9467-4431-bc52-40c0524ff214\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-km7m9"
Oct 31 13:23:52.865545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.865465    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m2tb\" (UniqueName: \"kubernetes.io/projected/e328741d-31fd-40db-8cbb-8c9b518f4cde-kube-api-access-5m2tb\") pod \"payload-processing-55fbbbf9dc-w9wjk\" (UID: \"e328741d-31fd-40db-8cbb-8c9b518f4cde\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w9wjk"
Oct 31 13:23:52.982162 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:52.982136    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-km7m9"
Oct 31 13:23:53.016272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:53.016252    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-w9wjk"
Oct 31 13:23:53.259780 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:53.259756    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-km7m9"]
Oct 31 13:23:53.261492 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:23:53.261466    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb129029_9467_4431_bc52_40c0524ff214.slice/crio-5b25a6271bda35136158689ea12e43c26af77809252cb2f074ffb0c11bf0d452 WatchSource:0}: Error finding container 5b25a6271bda35136158689ea12e43c26af77809252cb2f074ffb0c11bf0d452: Status 404 returned error can't find the container with id 5b25a6271bda35136158689ea12e43c26af77809252cb2f074ffb0c11bf0d452
Oct 31 13:23:53.299493 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:53.299468    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-w9wjk"]
Oct 31 13:23:53.302645 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:23:53.302606    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode328741d_31fd_40db_8cbb_8c9b518f4cde.slice/crio-9d09ca966d7c33ba24cbcac8fbd8eb792b999cffb1caf856f12b2ac6a27dc233 WatchSource:0}: Error finding container 9d09ca966d7c33ba24cbcac8fbd8eb792b999cffb1caf856f12b2ac6a27dc233: Status 404 returned error can't find the container with id 9d09ca966d7c33ba24cbcac8fbd8eb792b999cffb1caf856f12b2ac6a27dc233
Oct 31 13:23:53.705224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:53.705142    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-w9wjk" event={"ID":"e328741d-31fd-40db-8cbb-8c9b518f4cde","Type":"ContainerStarted","Data":"9d09ca966d7c33ba24cbcac8fbd8eb792b999cffb1caf856f12b2ac6a27dc233"}
Oct 31 13:23:53.706312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:53.706279    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-km7m9" event={"ID":"db129029-9467-4431-bc52-40c0524ff214","Type":"ContainerStarted","Data":"5b25a6271bda35136158689ea12e43c26af77809252cb2f074ffb0c11bf0d452"}
Oct 31 13:23:55.718395 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:55.718354    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-w9wjk" event={"ID":"e328741d-31fd-40db-8cbb-8c9b518f4cde","Type":"ContainerStarted","Data":"006134a16c03671afb52f921421e9ee723cc0111e2049f5b88a1b95784d166dd"}
Oct 31 13:23:55.718719 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:55.718452    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-w9wjk"
Oct 31 13:23:55.719822 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:55.719793    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-km7m9" event={"ID":"db129029-9467-4431-bc52-40c0524ff214","Type":"ContainerStarted","Data":"153b505ac70aa3d3c6bcc5ecba9654ac7561efdee5c21a0804cbdc9bc58f84a4"}
Oct 31 13:23:55.719922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:55.719912    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-km7m9"
Oct 31 13:23:55.739959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:55.739916    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-w9wjk" podStartSLOduration=1.485949861 podStartE2EDuration="3.739897347s" totalImagesPullingTime="2.253947486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:55.738045915 +0000 UTC m=+1233.136394539" watchObservedRunningTime="2026-10-31 13:23:55.739897347 +0000 UTC m=+1233.138245966"
Oct 31 13:23:55.758417 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:23:55.758379    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-km7m9" podStartSLOduration=1.423831429 podStartE2EDuration="3.758366853s" totalImagesPullingTime="2.334535424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:55.756279212 +0000 UTC m=+1233.154627837" watchObservedRunningTime="2026-10-31 13:23:55.758366853 +0000 UTC m=+1233.156715808"
Oct 31 13:24:02.701579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:02.701549    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"
Oct 31 13:24:02.976715 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:02.976680    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-58ff7d8b4-skjvb"]
Oct 31 13:24:02.985393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:02.985370    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-58ff7d8b4-skjvb"
Oct 31 13:24:02.990173 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:02.990147    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-58ff7d8b4-skjvb"]
Oct 31 13:24:03.140717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:03.140687    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30763b38-29e0-4998-843f-5471f4f621db-metrics-certs\") pod \"ai-gateway-controller-58ff7d8b4-skjvb\" (UID: \"30763b38-29e0-4998-843f-5471f4f621db\") " pod="opendatahub/ai-gateway-controller-58ff7d8b4-skjvb"
Oct 31 13:24:03.140859 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:03.140735    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4bwf\" (UniqueName: \"kubernetes.io/projected/30763b38-29e0-4998-843f-5471f4f621db-kube-api-access-k4bwf\") pod \"ai-gateway-controller-58ff7d8b4-skjvb\" (UID: \"30763b38-29e0-4998-843f-5471f4f621db\") " pod="opendatahub/ai-gateway-controller-58ff7d8b4-skjvb"
Oct 31 13:24:03.241982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:03.241923    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30763b38-29e0-4998-843f-5471f4f621db-metrics-certs\") pod \"ai-gateway-controller-58ff7d8b4-skjvb\" (UID: \"30763b38-29e0-4998-843f-5471f4f621db\") " pod="opendatahub/ai-gateway-controller-58ff7d8b4-skjvb"
Oct 31 13:24:03.241982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:03.241961    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4bwf\" (UniqueName: \"kubernetes.io/projected/30763b38-29e0-4998-843f-5471f4f621db-kube-api-access-k4bwf\") pod \"ai-gateway-controller-58ff7d8b4-skjvb\" (UID: \"30763b38-29e0-4998-843f-5471f4f621db\") " pod="opendatahub/ai-gateway-controller-58ff7d8b4-skjvb"
Oct 31 13:24:03.244421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:03.244395    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30763b38-29e0-4998-843f-5471f4f621db-metrics-certs\") pod \"ai-gateway-controller-58ff7d8b4-skjvb\" (UID: \"30763b38-29e0-4998-843f-5471f4f621db\") " pod="opendatahub/ai-gateway-controller-58ff7d8b4-skjvb"
Oct 31 13:24:03.255530 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:03.255510    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4bwf\" (UniqueName: \"kubernetes.io/projected/30763b38-29e0-4998-843f-5471f4f621db-kube-api-access-k4bwf\") pod \"ai-gateway-controller-58ff7d8b4-skjvb\" (UID: \"30763b38-29e0-4998-843f-5471f4f621db\") " pod="opendatahub/ai-gateway-controller-58ff7d8b4-skjvb"
Oct 31 13:24:03.300503 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:03.300483    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-58ff7d8b4-skjvb"
Oct 31 13:24:03.573385 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:03.573293    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-58ff7d8b4-skjvb"]
Oct 31 13:24:03.575855 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:24:03.575829    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30763b38_29e0_4998_843f_5471f4f621db.slice/crio-ddfea10e3e314ae38c56e95f620609edfea40744c42d71b2a6acdfa25455f757 WatchSource:0}: Error finding container ddfea10e3e314ae38c56e95f620609edfea40744c42d71b2a6acdfa25455f757: Status 404 returned error can't find the container with id ddfea10e3e314ae38c56e95f620609edfea40744c42d71b2a6acdfa25455f757
Oct 31 13:24:03.759821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:03.759791    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-58ff7d8b4-skjvb" event={"ID":"30763b38-29e0-4998-843f-5471f4f621db","Type":"ContainerStarted","Data":"ddfea10e3e314ae38c56e95f620609edfea40744c42d71b2a6acdfa25455f757"}
Oct 31 13:24:04.765501 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:04.765465    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-58ff7d8b4-skjvb" event={"ID":"30763b38-29e0-4998-843f-5471f4f621db","Type":"ContainerStarted","Data":"9907c156581b5c7f671a0775f9df5f0f0faff0209464dece41b6e34ab3bf2151"}
Oct 31 13:24:04.765944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:04.765607    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-58ff7d8b4-skjvb"
Oct 31 13:24:04.786821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:04.786781    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-58ff7d8b4-skjvb" podStartSLOduration=2.222165428 podStartE2EDuration="2.786770458s" totalImagesPullingTime="564.60503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:04.785342331 +0000 UTC m=+1242.183690955" watchObservedRunningTime="2026-10-31 13:24:04.786770458 +0000 UTC m=+1242.185119118"
Oct 31 13:24:06.322136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:06.322105    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:06.727833 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:06.727809    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-w9wjk"
Oct 31 13:24:06.728025 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:06.727892    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-km7m9"
Oct 31 13:24:12.139021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:12.138970    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:15.774616 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:15.774587    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-58ff7d8b4-skjvb"
Oct 31 13:24:15.827171 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:15.827134    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"]
Oct 31 13:24:15.827437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:15.827385    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2" podUID="10f71b0f-cc6f-4d6a-add0-56ce2113fb47" containerName="manager" containerID="cri-o://41af66cf18cf550907843a6b03391f3a9aadf4e410deb586643f63be00144622" gracePeriod=30
Oct 31 13:24:16.075039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:16.075016    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"
Oct 31 13:24:16.251747 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:16.251716    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10f71b0f-cc6f-4d6a-add0-56ce2113fb47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10f71b0f-cc6f-4d6a-add0-56ce2113fb47-metrics-certs\") pod \"10f71b0f-cc6f-4d6a-add0-56ce2113fb47\" (UID: \"10f71b0f-cc6f-4d6a-add0-56ce2113fb47\") "
Oct 31 13:24:16.251982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:16.251756    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f71b0f-cc6f-4d6a-add0-56ce2113fb47-kube-api-access-pnrs9\" (UniqueName: \"kubernetes.io/projected/10f71b0f-cc6f-4d6a-add0-56ce2113fb47-kube-api-access-pnrs9\") pod \"10f71b0f-cc6f-4d6a-add0-56ce2113fb47\" (UID: \"10f71b0f-cc6f-4d6a-add0-56ce2113fb47\") "
Oct 31 13:24:16.254152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:16.254118    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10f71b0f-cc6f-4d6a-add0-56ce2113fb47-metrics-certs" pod "10f71b0f-cc6f-4d6a-add0-56ce2113fb47" (UID: "10f71b0f-cc6f-4d6a-add0-56ce2113fb47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:24:16.254276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:16.254227    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f71b0f-cc6f-4d6a-add0-56ce2113fb47-kube-api-access-pnrs9" pod "10f71b0f-cc6f-4d6a-add0-56ce2113fb47" (UID: "10f71b0f-cc6f-4d6a-add0-56ce2113fb47"). InnerVolumeSpecName "kube-api-access-pnrs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:24:16.352766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:16.352710    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10f71b0f-cc6f-4d6a-add0-56ce2113fb47-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:24:16.352766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:16.352735    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnrs9\" (UniqueName: \"kubernetes.io/projected/10f71b0f-cc6f-4d6a-add0-56ce2113fb47-kube-api-access-pnrs9\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:24:16.821658 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:16.821626    4428 generic.go:356] "Generic (PLEG): container finished" podID="10f71b0f-cc6f-4d6a-add0-56ce2113fb47" containerID="41af66cf18cf550907843a6b03391f3a9aadf4e410deb586643f63be00144622" exitCode=0
Oct 31 13:24:16.822107 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:16.821686    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"
Oct 31 13:24:16.822107 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:16.821707    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2" event={"ID":"10f71b0f-cc6f-4d6a-add0-56ce2113fb47","Type":"ContainerDied","Data":"41af66cf18cf550907843a6b03391f3a9aadf4e410deb586643f63be00144622"}
Oct 31 13:24:16.822107 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:16.821773    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-qtxk2" event={"ID":"10f71b0f-cc6f-4d6a-add0-56ce2113fb47","Type":"ContainerDied","Data":"c21ca4d8626ecb447b4d33732657648de5ad2038ccdb3092a34fd17fdfcbd6ae"}
Oct 31 13:24:16.822107 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:16.821807    4428 scope.go:122] "RemoveContainer" containerID="41af66cf18cf550907843a6b03391f3a9aadf4e410deb586643f63be00144622"
Oct 31 13:24:16.830260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:16.829970    4428 scope.go:122] "RemoveContainer" containerID="41af66cf18cf550907843a6b03391f3a9aadf4e410deb586643f63be00144622"
Oct 31 13:24:16.830314 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:24:16.830265    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41af66cf18cf550907843a6b03391f3a9aadf4e410deb586643f63be00144622\": container with ID starting with 41af66cf18cf550907843a6b03391f3a9aadf4e410deb586643f63be00144622 not found: ID does not exist" containerID="41af66cf18cf550907843a6b03391f3a9aadf4e410deb586643f63be00144622"
Oct 31 13:24:16.830314 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:16.830288    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41af66cf18cf550907843a6b03391f3a9aadf4e410deb586643f63be00144622"} err="failed to get container status \"41af66cf18cf550907843a6b03391f3a9aadf4e410deb586643f63be00144622\": rpc error: code = NotFound desc = could not find container \"41af66cf18cf550907843a6b03391f3a9aadf4e410deb586643f63be00144622\": container with ID starting with 41af66cf18cf550907843a6b03391f3a9aadf4e410deb586643f63be00144622 not found: ID does not exist"
Oct 31 13:24:16.841180 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:16.841156    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"]
Oct 31 13:24:16.845746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:16.845727    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-qtxk2"]
Oct 31 13:24:17.581164 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:17.581126    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:18.738801 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:24:18.738766    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:24:18.753524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:18.753498    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f71b0f-cc6f-4d6a-add0-56ce2113fb47" path="/var/lib/kubelet/pods/10f71b0f-cc6f-4d6a-add0-56ce2113fb47/volumes"
Oct 31 13:24:19.064643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:19.064576    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:19.708248 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:19.708211    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:19.866182 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:19.866147    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:21.252547 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:21.252507    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:21.404839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:21.404812    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:38.910055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:38.910026    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:42.516287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.516249    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:42.672348 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.672320    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:42.901503 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.901429    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"]
Oct 31 13:24:42.902002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.901983    4428 cpu_manager.go:414] "containerMap: removing container" podUID="10f71b0f-cc6f-4d6a-add0-56ce2113fb47" containerName="manager"
Oct 31 13:24:42.902086 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.902005    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f71b0f-cc6f-4d6a-add0-56ce2113fb47" containerName="manager"
Oct 31 13:24:42.902086 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.902079    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f71b0f-cc6f-4d6a-add0-56ce2113fb47" containerName="manager"
Oct 31 13:24:42.905106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.905088    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:42.915725 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.915701    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-6784d6-openshift-default-dockercfg-xbv85\""
Oct 31 13:24:42.919310 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.919287    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"]
Oct 31 13:24:42.972175 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.972149    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:42.972297 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.972185    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:42.972297 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.972205    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:42.972297 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.972226    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:42.972297 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.972278    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:42.972459 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.972439    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:42.972495 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.972471    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:42.972528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.972505    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:42.972578 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:42.972566    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw8nc\" (UniqueName: \"kubernetes.io/projected/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-kube-api-access-rw8nc\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.046006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.045971    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:43.074049 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.074014    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.074198 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.074052    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.074198 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.074092    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.074198 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.074122    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.074198 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.074169    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.074465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.074230    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw8nc\" (UniqueName: \"kubernetes.io/projected/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-kube-api-access-rw8nc\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.074465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.074318    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.074465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.074363    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.074465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.074401    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.074465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.074445    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.074728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.074531    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.074836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.074811    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.075010 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.074990    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.075090 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.075074    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.076421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.076403    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.076625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.076610    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.090211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.090185    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.094644 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.094621    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw8nc\" (UniqueName: \"kubernetes.io/projected/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-kube-api-access-rw8nc\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.135210 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.135177    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"]
Oct 31 13:24:43.141625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.140276    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.143043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.143019    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"]
Oct 31 13:24:43.147619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.147596    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-c82140-openshift-default-dockercfg-bmc79\""
Oct 31 13:24:43.175220 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.175159    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.175220 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.175192    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.175220 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.175216    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.175422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.175322    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.175422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.175371    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.175504 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.175422    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.175504 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.175448    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.175504 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.175467    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.175658 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.175600    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cml8j\" (UniqueName: \"kubernetes.io/projected/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-kube-api-access-cml8j\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.219448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.219424    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:43.255786 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.255756    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:43.276410 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.276384    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.276490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.276430    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.276490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.276456    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.276602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.276487    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.276602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.276513    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.276602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.276556    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.276602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.276596    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.276815 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.276617    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.276815 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.276665    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cml8j\" (UniqueName: \"kubernetes.io/projected/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-kube-api-access-cml8j\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.277022 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.276995    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.277259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.277237    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.277578 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.277550    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.277665 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.277647    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.278366 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.278336    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.280786 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.280758    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.281855 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.281826    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.296639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.296608    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cml8j\" (UniqueName: \"kubernetes.io/projected/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-kube-api-access-cml8j\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.300285 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.300260    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.342626 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.342596    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"]
Oct 31 13:24:43.347532 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.347505    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.357960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.354955    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-aabeef-openshift-default-dockercfg-wlnkc\""
Oct 31 13:24:43.357960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.357914    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"]
Oct 31 13:24:43.377914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.377886    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.378026 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.377991    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.378067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.378054    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.378105 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.378075    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.378105 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.378100    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.378176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.378157    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsxvp\" (UniqueName: \"kubernetes.io/projected/d5e6e17a-543e-4373-ad93-e0ed60c74334-kube-api-access-dsxvp\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.378337 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.378323    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.378376 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.378348    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.378407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.378376    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e6e17a-543e-4373-ad93-e0ed60c74334-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.417796 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.417764    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"]
Oct 31 13:24:43.421911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.421892    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.435650 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.435623    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"]
Oct 31 13:24:43.443135 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.443097    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:43.454182 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.453839    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-5286ea-openshift-default-dockercfg-rsjlf\""
Oct 31 13:24:43.456206 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.456184    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:43.479640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.479611    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.479764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.479660    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.479764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.479686    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6002eb05-9237-48f2-972a-448248703ca4-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.479764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.479722    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.479934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.479762    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e6e17a-543e-4373-ad93-e0ed60c74334-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.479934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.479789    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6002eb05-9237-48f2-972a-448248703ca4-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.479934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.479831    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6002eb05-9237-48f2-972a-448248703ca4-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.479934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.479897    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.479934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.479925    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.480120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.479949    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcsjt\" (UniqueName: \"kubernetes.io/projected/6002eb05-9237-48f2-972a-448248703ca4-kube-api-access-qcsjt\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.480120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.479983    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.480120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.480027    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.480120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.480062    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.480268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.480150    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.480268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.480184    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.480268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.480213    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.480268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.480242    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.480268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.480264    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsxvp\" (UniqueName: \"kubernetes.io/projected/d5e6e17a-543e-4373-ad93-e0ed60c74334-kube-api-access-dsxvp\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.480941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.480860    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.481146 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.481119    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.481294 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.481222    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.481524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.481505    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.481702 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.481675    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e6e17a-543e-4373-ad93-e0ed60c74334-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.483004 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.482982    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.483329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.483308    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.504801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.504776    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.505905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.505843    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsxvp\" (UniqueName: \"kubernetes.io/projected/d5e6e17a-543e-4373-ad93-e0ed60c74334-kube-api-access-dsxvp\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.514584 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.514383    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"]
Oct 31 13:24:43.531515 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:24:43.531076    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1667cc7b_35eb_4c92_9fa7_239aaa5990d1.slice/crio-bd11ad3de22abdaa84ae8db21fcd82ff0c72f34cc52bb696260ad04dccfaaa77 WatchSource:0}: Error finding container bd11ad3de22abdaa84ae8db21fcd82ff0c72f34cc52bb696260ad04dccfaaa77: Status 404 returned error can't find the container with id bd11ad3de22abdaa84ae8db21fcd82ff0c72f34cc52bb696260ad04dccfaaa77
Oct 31 13:24:43.534350 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.534321    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:24:43.534471 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.534402    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:24:43.534539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.534457    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:24:43.580965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.580934    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6002eb05-9237-48f2-972a-448248703ca4-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.581080 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.580980    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6002eb05-9237-48f2-972a-448248703ca4-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.581080 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.581029    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.581080 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.581056    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcsjt\" (UniqueName: \"kubernetes.io/projected/6002eb05-9237-48f2-972a-448248703ca4-kube-api-access-qcsjt\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.581249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.581094    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.581249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.581145    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.581249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.581225    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.581392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.581312    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6002eb05-9237-48f2-972a-448248703ca4-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.581392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.581349    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.581781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.581609    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.581781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.581686    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.581781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.581704    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.582166 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.582144    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.583802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.583777    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6002eb05-9237-48f2-972a-448248703ca4-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.584446 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.584417    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.584896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.584845    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6002eb05-9237-48f2-972a-448248703ca4-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.602934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.602912    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6002eb05-9237-48f2-972a-448248703ca4-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.603172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.603150    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcsjt\" (UniqueName: \"kubernetes.io/projected/6002eb05-9237-48f2-972a-448248703ca4-kube-api-access-qcsjt\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.624548 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.624524    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:43.664133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.664109    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:43.746899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.745064    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:43.787266 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.787183    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"]
Oct 31 13:24:43.814925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.814780    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:43.827446 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:24:43.827408    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd4370a_9ed2_47aa_b9cf_8a6300944bd6.slice/crio-20b7ff07f0c82c988de877083d42f953c75932037f2821e2c84090d2e24dd9b8 WatchSource:0}: Error finding container 20b7ff07f0c82c988de877083d42f953c75932037f2821e2c84090d2e24dd9b8: Status 404 returned error can't find the container with id 20b7ff07f0c82c988de877083d42f953c75932037f2821e2c84090d2e24dd9b8
Oct 31 13:24:43.830954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.830196    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:24:43.830954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.830292    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:24:43.830954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.830340    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:24:43.942242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.942208    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" event={"ID":"1667cc7b-35eb-4c92-9fa7-239aaa5990d1","Type":"ContainerStarted","Data":"71bc7b5aabe8ff9a363889f5436413ba9693154a01dcd88ae0d9437b88bf20f4"}
Oct 31 13:24:43.942813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.942788    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" event={"ID":"1667cc7b-35eb-4c92-9fa7-239aaa5990d1","Type":"ContainerStarted","Data":"bd11ad3de22abdaa84ae8db21fcd82ff0c72f34cc52bb696260ad04dccfaaa77"}
Oct 31 13:24:43.945986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.944613    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj" event={"ID":"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6","Type":"ContainerStarted","Data":"4bc53ab41306a1cd1c8e0ce93d07ffc45d20dcc0f3d3676ef4e3185c5fcfbe05"}
Oct 31 13:24:43.945986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.944657    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj" event={"ID":"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6","Type":"ContainerStarted","Data":"20b7ff07f0c82c988de877083d42f953c75932037f2821e2c84090d2e24dd9b8"}
Oct 31 13:24:43.968170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.968126    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" podStartSLOduration=1.9681151940000001 podStartE2EDuration="1.968115194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:43.967149647 +0000 UTC m=+1281.365498277" watchObservedRunningTime="2026-10-31 13:24:43.968115194 +0000 UTC m=+1281.366463812"
Oct 31 13:24:43.995750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:43.994932    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj" podStartSLOduration=0.994918236 podStartE2EDuration="994.918236ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:43.993311102 +0000 UTC m=+1281.391659726" watchObservedRunningTime="2026-10-31 13:24:43.994918236 +0000 UTC m=+1281.393266860"
Oct 31 13:24:44.004760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.004690    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:44.005800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.005781    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"]
Oct 31 13:24:44.024489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.024456    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:24:44.024576 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.024539    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:24:44.024619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.024577    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:24:44.114937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.114907    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"]
Oct 31 13:24:44.152657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.152623    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:24:44.152787 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.152713    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:24:44.152787 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.152755    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:24:44.220568 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.220544    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:44.227960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.227936    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:44.456833 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.456745    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:44.462025 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.461998    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:44.950888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.950823    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld" event={"ID":"6002eb05-9237-48f2-972a-448248703ca4","Type":"ContainerStarted","Data":"86f4b0c34be4808cb41009e58c6fb3a9ddd7aab94ae09bd4c85399f1e7fa1580"}
Oct 31 13:24:44.951315 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.950917    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld" event={"ID":"6002eb05-9237-48f2-972a-448248703ca4","Type":"ContainerStarted","Data":"052cf1c2b9fd3545859b3549b83b49884128bbba2cc269822c7e1b99616cfe9f"}
Oct 31 13:24:44.952571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.952544    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj" event={"ID":"d5e6e17a-543e-4373-ad93-e0ed60c74334","Type":"ContainerStarted","Data":"f4c42ef64539004b74cd50af897ac2cedec54f5db0d8bb70a148211da1ad9041"}
Oct 31 13:24:44.952722 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.952590    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj" event={"ID":"d5e6e17a-543e-4373-ad93-e0ed60c74334","Type":"ContainerStarted","Data":"bf81813fa6df6a74c36fd39dde7f3ed59cda1af5fc71da5afb5ab9cd3ab775b7"}
Oct 31 13:24:44.952926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.952906    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:44.953463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.952956    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:44.953712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.953693    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:24:44.953759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.953747    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:24:44.976452 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:44.976397    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld" podStartSLOduration=1.976385374 podStartE2EDuration="1.976385374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:44.974242374 +0000 UTC m=+1282.372591041" watchObservedRunningTime="2026-10-31 13:24:44.976385374 +0000 UTC m=+1282.374733997"
Oct 31 13:24:45.052230 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:45.052169    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj" podStartSLOduration=2.052149868 podStartE2EDuration="2.052149868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:45.050545007 +0000 UTC m=+1282.448893632" watchObservedRunningTime="2026-10-31 13:24:45.052149868 +0000 UTC m=+1282.450498509"
Oct 31 13:24:45.664537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:45.664507    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:45.669440 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:45.669416    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:45.746472 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:45.746444    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:45.752749 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:45.752726    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:45.957543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:45.957451    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:45.957543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:45.957493    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:45.958035 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:45.957921    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:24:45.958363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:45.958349    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:24:46.334523 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:46.334493    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:46.501452 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:46.501420    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:46.660177 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:46.660109    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:47.123534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:47.123503    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:47.351082 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:47.351052    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:47.532702 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:47.532669    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:47.622689 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:47.622654    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"]
Oct 31 13:24:47.626632 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:47.626610    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"
Oct 31 13:24:47.639499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:47.639472    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"]
Oct 31 13:24:47.668630 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:47.668579    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-6784d6-dockercfg-t6zsz\""
Oct 31 13:24:47.693496 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:47.693468    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:47.728823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:47.728797    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-6784d6\""
Oct 31 13:24:47.729493 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:47.729453    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpmh7\" (UniqueName: \"kubernetes.io/projected/4ea944d0-8db8-4ae4-8e37-f04879e47a6a-kube-api-access-kpmh7\") pod \"payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t\" (UID: \"4ea944d0-8db8-4ae4-8e37-f04879e47a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"
Oct 31 13:24:47.729776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:47.729752    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea944d0-8db8-4ae4-8e37-f04879e47a6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t\" (UID: \"4ea944d0-8db8-4ae4-8e37-f04879e47a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"
Oct 31 13:24:47.831041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:47.830969    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpmh7\" (UniqueName: \"kubernetes.io/projected/4ea944d0-8db8-4ae4-8e37-f04879e47a6a-kube-api-access-kpmh7\") pod \"payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t\" (UID: \"4ea944d0-8db8-4ae4-8e37-f04879e47a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"
Oct 31 13:24:47.831041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:47.831008    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea944d0-8db8-4ae4-8e37-f04879e47a6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t\" (UID: \"4ea944d0-8db8-4ae4-8e37-f04879e47a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"
Oct 31 13:24:47.831673 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:47.831646    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea944d0-8db8-4ae4-8e37-f04879e47a6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t\" (UID: \"4ea944d0-8db8-4ae4-8e37-f04879e47a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"
Oct 31 13:24:47.844169 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:47.844144    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpmh7\" (UniqueName: \"kubernetes.io/projected/4ea944d0-8db8-4ae4-8e37-f04879e47a6a-kube-api-access-kpmh7\") pod \"payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t\" (UID: \"4ea944d0-8db8-4ae4-8e37-f04879e47a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"
Oct 31 13:24:47.949123 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:47.949092    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"
Oct 31 13:24:48.133484 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.133413    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"]
Oct 31 13:24:48.138380 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.138360    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"
Oct 31 13:24:48.146403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.146361    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-c82140-dockercfg-7xrq5\""
Oct 31 13:24:48.146714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.146692    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-c82140\""
Oct 31 13:24:48.147809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.147789    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"]
Oct 31 13:24:48.158953 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.158912    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:48.236496 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.236457    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwj95\" (UniqueName: \"kubernetes.io/projected/ee893209-7839-4a39-8aaa-17dc0f90d8d7-kube-api-access-cwj95\") pod \"payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb\" (UID: \"ee893209-7839-4a39-8aaa-17dc0f90d8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"
Oct 31 13:24:48.236661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.236597    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee893209-7839-4a39-8aaa-17dc0f90d8d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb\" (UID: \"ee893209-7839-4a39-8aaa-17dc0f90d8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"
Oct 31 13:24:48.254631 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.254609    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"]
Oct 31 13:24:48.256423 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:24:48.256383    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea944d0_8db8_4ae4_8e37_f04879e47a6a.slice/crio-66125c71ebc736d54d2af634dcf783fa9400c4d36b198ae1971d3b23682f7929 WatchSource:0}: Error finding container 66125c71ebc736d54d2af634dcf783fa9400c4d36b198ae1971d3b23682f7929: Status 404 returned error can't find the container with id 66125c71ebc736d54d2af634dcf783fa9400c4d36b198ae1971d3b23682f7929
Oct 31 13:24:48.352901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.338420    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee893209-7839-4a39-8aaa-17dc0f90d8d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb\" (UID: \"ee893209-7839-4a39-8aaa-17dc0f90d8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"
Oct 31 13:24:48.352901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.338584    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwj95\" (UniqueName: \"kubernetes.io/projected/ee893209-7839-4a39-8aaa-17dc0f90d8d7-kube-api-access-cwj95\") pod \"payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb\" (UID: \"ee893209-7839-4a39-8aaa-17dc0f90d8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"
Oct 31 13:24:48.352901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.343476    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee893209-7839-4a39-8aaa-17dc0f90d8d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb\" (UID: \"ee893209-7839-4a39-8aaa-17dc0f90d8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"
Oct 31 13:24:48.364942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.359029    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:48.377995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.372162    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwj95\" (UniqueName: \"kubernetes.io/projected/ee893209-7839-4a39-8aaa-17dc0f90d8d7-kube-api-access-cwj95\") pod \"payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb\" (UID: \"ee893209-7839-4a39-8aaa-17dc0f90d8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"
Oct 31 13:24:48.453806 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.453733    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"
Oct 31 13:24:48.622953 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.622918    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:48.738520 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.738492    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"]
Oct 31 13:24:48.739276 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:24:48.739251    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee893209_7839_4a39_8aaa_17dc0f90d8d7.slice/crio-69c5a29bb00ed250e35a8e5272a6e0b8e270197674189c4e836d5e76780c0fe1 WatchSource:0}: Error finding container 69c5a29bb00ed250e35a8e5272a6e0b8e270197674189c4e836d5e76780c0fe1: Status 404 returned error can't find the container with id 69c5a29bb00ed250e35a8e5272a6e0b8e270197674189c4e836d5e76780c0fe1
Oct 31 13:24:48.743427 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:24:48.743400    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:24:48.781618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.781596    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"]
Oct 31 13:24:48.786451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.786431    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"
Oct 31 13:24:48.793073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.793038    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-d85d82\""
Oct 31 13:24:48.793172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.793113    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-d85d82-dockercfg-vwfrq\""
Oct 31 13:24:48.795257 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.795204    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"]
Oct 31 13:24:48.801722 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.801688    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:48.844208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.844182    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf16863d-965a-458c-bf0d-966d50096325-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8\" (UID: \"bf16863d-965a-458c-bf0d-966d50096325\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"
Oct 31 13:24:48.844370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.844260    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96wjp\" (UniqueName: \"kubernetes.io/projected/bf16863d-965a-458c-bf0d-966d50096325-kube-api-access-96wjp\") pod \"payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8\" (UID: \"bf16863d-965a-458c-bf0d-966d50096325\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"
Oct 31 13:24:48.954407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.954370    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf16863d-965a-458c-bf0d-966d50096325-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8\" (UID: \"bf16863d-965a-458c-bf0d-966d50096325\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"
Oct 31 13:24:48.954625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.954442    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96wjp\" (UniqueName: \"kubernetes.io/projected/bf16863d-965a-458c-bf0d-966d50096325-kube-api-access-96wjp\") pod \"payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8\" (UID: \"bf16863d-965a-458c-bf0d-966d50096325\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"
Oct 31 13:24:48.957676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.955294    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf16863d-965a-458c-bf0d-966d50096325-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8\" (UID: \"bf16863d-965a-458c-bf0d-966d50096325\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"
Oct 31 13:24:48.973838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.973803    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96wjp\" (UniqueName: \"kubernetes.io/projected/bf16863d-965a-458c-bf0d-966d50096325-kube-api-access-96wjp\") pod \"payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8\" (UID: \"bf16863d-965a-458c-bf0d-966d50096325\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"
Oct 31 13:24:48.980509 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.980476    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:48.985305 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.985262    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb" event={"ID":"ee893209-7839-4a39-8aaa-17dc0f90d8d7","Type":"ContainerStarted","Data":"bc0ed2137f062dbff31cbab5f0d1f1804312409d8a3ed243ae582fa39c7b2e44"}
Oct 31 13:24:48.985454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.985325    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb" event={"ID":"ee893209-7839-4a39-8aaa-17dc0f90d8d7","Type":"ContainerStarted","Data":"69c5a29bb00ed250e35a8e5272a6e0b8e270197674189c4e836d5e76780c0fe1"}
Oct 31 13:24:48.985454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.985417    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"
Oct 31 13:24:48.987548 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.987417    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t" event={"ID":"4ea944d0-8db8-4ae4-8e37-f04879e47a6a","Type":"ContainerStarted","Data":"c0d48616b67d2a1af035c4edfebc61fba790923710cf6fefa99141887c42d7aa"}
Oct 31 13:24:48.987548 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.987465    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t" event={"ID":"4ea944d0-8db8-4ae4-8e37-f04879e47a6a","Type":"ContainerStarted","Data":"66125c71ebc736d54d2af634dcf783fa9400c4d36b198ae1971d3b23682f7929"}
Oct 31 13:24:48.988054 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:48.988022    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"
Oct 31 13:24:49.009970 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.009932    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb" podStartSLOduration=1.009921859 podStartE2EDuration="1.009921859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:49.008449388 +0000 UTC m=+1286.406798009" watchObservedRunningTime="2026-10-31 13:24:49.009921859 +0000 UTC m=+1286.408270482"
Oct 31 13:24:49.028157 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.028112    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t" podStartSLOduration=2.028101561 podStartE2EDuration="2.028101561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:49.026969672 +0000 UTC m=+1286.425318298" watchObservedRunningTime="2026-10-31 13:24:49.028101561 +0000 UTC m=+1286.426450185"
Oct 31 13:24:49.103343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.103317    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"
Oct 31 13:24:49.215504 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.215470    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:49.381767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.381670    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:49.410991 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.410962    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"]
Oct 31 13:24:49.415499 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:24:49.415467    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf16863d_965a_458c_bf0d_966d50096325.slice/crio-87a8ff8cb16203b02fd4358bf3466dfed1ef82953458da70b0b7a58d1cc9fc72 WatchSource:0}: Error finding container 87a8ff8cb16203b02fd4358bf3466dfed1ef82953458da70b0b7a58d1cc9fc72: Status 404 returned error can't find the container with id 87a8ff8cb16203b02fd4358bf3466dfed1ef82953458da70b0b7a58d1cc9fc72
Oct 31 13:24:49.542661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.542619    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:49.772793 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.772765    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"]
Oct 31 13:24:49.776376 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.776358    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"
Oct 31 13:24:49.783214 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.783195    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-8b2f0a\""
Oct 31 13:24:49.792460 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.792436    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"]
Oct 31 13:24:49.810922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.810827    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:49.878167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.878140    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859ad249-88cd-4054-a411-f0847db07b12-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs\" (UID: \"859ad249-88cd-4054-a411-f0847db07b12\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"
Oct 31 13:24:49.878328 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.878209    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl46z\" (UniqueName: \"kubernetes.io/projected/859ad249-88cd-4054-a411-f0847db07b12-kube-api-access-kl46z\") pod \"payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs\" (UID: \"859ad249-88cd-4054-a411-f0847db07b12\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"
Oct 31 13:24:49.967227 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.967202    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:49.981527 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.981496    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl46z\" (UniqueName: \"kubernetes.io/projected/859ad249-88cd-4054-a411-f0847db07b12-kube-api-access-kl46z\") pod \"payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs\" (UID: \"859ad249-88cd-4054-a411-f0847db07b12\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"
Oct 31 13:24:49.981660 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.981643    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859ad249-88cd-4054-a411-f0847db07b12-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs\" (UID: \"859ad249-88cd-4054-a411-f0847db07b12\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"
Oct 31 13:24:49.982537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.982516    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859ad249-88cd-4054-a411-f0847db07b12-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs\" (UID: \"859ad249-88cd-4054-a411-f0847db07b12\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"
Oct 31 13:24:50.016260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:49.997088    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl46z\" (UniqueName: \"kubernetes.io/projected/859ad249-88cd-4054-a411-f0847db07b12-kube-api-access-kl46z\") pod \"payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs\" (UID: \"859ad249-88cd-4054-a411-f0847db07b12\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"
Oct 31 13:24:50.038941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.024826    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8" event={"ID":"bf16863d-965a-458c-bf0d-966d50096325","Type":"ContainerStarted","Data":"ced40dcb0459fd8ec4eb8c8770c524bf1b11f317fd7c9878172b5e4e1cad1409"}
Oct 31 13:24:50.038941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.024904    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8" event={"ID":"bf16863d-965a-458c-bf0d-966d50096325","Type":"ContainerStarted","Data":"87a8ff8cb16203b02fd4358bf3466dfed1ef82953458da70b0b7a58d1cc9fc72"}
Oct 31 13:24:50.038941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.025452    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"
Oct 31 13:24:50.089552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.089489    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"
Oct 31 13:24:50.216819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.216776    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:50.255483 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.255435    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8" podStartSLOduration=2.255421651 podStartE2EDuration="2.255421651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:50.052792698 +0000 UTC m=+1287.451141324" watchObservedRunningTime="2026-10-31 13:24:50.255421651 +0000 UTC m=+1287.653770308"
Oct 31 13:24:50.255621 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.255551    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"]
Oct 31 13:24:50.265740 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.265687    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"
Oct 31 13:24:50.270542 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.270516    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-aabeef\""
Oct 31 13:24:50.270826 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.270795    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-aabeef-dockercfg-hw68p\""
Oct 31 13:24:50.274166 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.274138    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"]
Oct 31 13:24:50.303489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.303419    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"]
Oct 31 13:24:50.307816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.307795    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"
Oct 31 13:24:50.331437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.331412    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"]
Oct 31 13:24:50.391896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.390599    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2q5t\" (UniqueName: \"kubernetes.io/projected/b570ecdd-7dda-4021-966d-a36080ef5293-kube-api-access-k2q5t\") pod \"payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w\" (UID: \"b570ecdd-7dda-4021-966d-a36080ef5293\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"
Oct 31 13:24:50.391896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.390665    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv4vp\" (UniqueName: \"kubernetes.io/projected/7348227d-79bb-4bc8-b6ae-f4e318dcc703-kube-api-access-rv4vp\") pod \"payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp\" (UID: \"7348227d-79bb-4bc8-b6ae-f4e318dcc703\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"
Oct 31 13:24:50.391896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.390715    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b570ecdd-7dda-4021-966d-a36080ef5293-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w\" (UID: \"b570ecdd-7dda-4021-966d-a36080ef5293\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"
Oct 31 13:24:50.391896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.390778    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7348227d-79bb-4bc8-b6ae-f4e318dcc703-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp\" (UID: \"7348227d-79bb-4bc8-b6ae-f4e318dcc703\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"
Oct 31 13:24:50.394741 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.394692    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:50.440167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.440134    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"]
Oct 31 13:24:50.446257 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:24:50.446231    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859ad249_88cd_4054_a411_f0847db07b12.slice/crio-b982a932440f42b347622d9ad123e4cc806e6ad7718d41ae13194f59f1a4888c WatchSource:0}: Error finding container b982a932440f42b347622d9ad123e4cc806e6ad7718d41ae13194f59f1a4888c: Status 404 returned error can't find the container with id b982a932440f42b347622d9ad123e4cc806e6ad7718d41ae13194f59f1a4888c
Oct 31 13:24:50.492066 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.492037    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2q5t\" (UniqueName: \"kubernetes.io/projected/b570ecdd-7dda-4021-966d-a36080ef5293-kube-api-access-k2q5t\") pod \"payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w\" (UID: \"b570ecdd-7dda-4021-966d-a36080ef5293\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"
Oct 31 13:24:50.492170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.492086    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv4vp\" (UniqueName: \"kubernetes.io/projected/7348227d-79bb-4bc8-b6ae-f4e318dcc703-kube-api-access-rv4vp\") pod \"payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp\" (UID: \"7348227d-79bb-4bc8-b6ae-f4e318dcc703\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"
Oct 31 13:24:50.492170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.492118    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b570ecdd-7dda-4021-966d-a36080ef5293-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w\" (UID: \"b570ecdd-7dda-4021-966d-a36080ef5293\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"
Oct 31 13:24:50.492170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.492158    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7348227d-79bb-4bc8-b6ae-f4e318dcc703-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp\" (UID: \"7348227d-79bb-4bc8-b6ae-f4e318dcc703\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"
Oct 31 13:24:50.492815 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.492798    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7348227d-79bb-4bc8-b6ae-f4e318dcc703-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp\" (UID: \"7348227d-79bb-4bc8-b6ae-f4e318dcc703\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"
Oct 31 13:24:50.492849 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.492810    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b570ecdd-7dda-4021-966d-a36080ef5293-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w\" (UID: \"b570ecdd-7dda-4021-966d-a36080ef5293\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"
Oct 31 13:24:50.505320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.505300    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2q5t\" (UniqueName: \"kubernetes.io/projected/b570ecdd-7dda-4021-966d-a36080ef5293-kube-api-access-k2q5t\") pod \"payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w\" (UID: \"b570ecdd-7dda-4021-966d-a36080ef5293\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"
Oct 31 13:24:50.505477 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.505462    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv4vp\" (UniqueName: \"kubernetes.io/projected/7348227d-79bb-4bc8-b6ae-f4e318dcc703-kube-api-access-rv4vp\") pod \"payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp\" (UID: \"7348227d-79bb-4bc8-b6ae-f4e318dcc703\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"
Oct 31 13:24:50.585688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.585626    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"
Oct 31 13:24:50.620503 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.620475    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"
Oct 31 13:24:50.718053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.718025    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:50.743371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.743338    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"]
Oct 31 13:24:50.748993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.748972    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"
Oct 31 13:24:50.765934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.762057    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-5286ea\""
Oct 31 13:24:50.771850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.771826    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"]
Oct 31 13:24:50.794895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.794839    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-5286ea-dockercfg-7xhhv\""
Oct 31 13:24:50.884040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.884019    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"]
Oct 31 13:24:50.885580 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:24:50.885553    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb570ecdd_7dda_4021_966d_a36080ef5293.slice/crio-ebe130c912b8c994155ed78b4b8a9c6573323beb07de02e9ddb648d56872714f WatchSource:0}: Error finding container ebe130c912b8c994155ed78b4b8a9c6573323beb07de02e9ddb648d56872714f: Status 404 returned error can't find the container with id ebe130c912b8c994155ed78b4b8a9c6573323beb07de02e9ddb648d56872714f
Oct 31 13:24:50.889258 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.889227    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:50.899480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.896685    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92cd891-8b41-4511-9fb0-d724cb1caaf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5\" (UID: \"c92cd891-8b41-4511-9fb0-d724cb1caaf5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"
Oct 31 13:24:50.899480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.896721    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwp8w\" (UniqueName: \"kubernetes.io/projected/c92cd891-8b41-4511-9fb0-d724cb1caaf5-kube-api-access-dwp8w\") pod \"payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5\" (UID: \"c92cd891-8b41-4511-9fb0-d724cb1caaf5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"
Oct 31 13:24:50.970000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.969499    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"]
Oct 31 13:24:50.977175 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:24:50.975365    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7348227d_79bb_4bc8_b6ae_f4e318dcc703.slice/crio-370c3b8f8b496f2d21e7c7280f23424b80657dbbc6bd48a8493caf7f1b11004e WatchSource:0}: Error finding container 370c3b8f8b496f2d21e7c7280f23424b80657dbbc6bd48a8493caf7f1b11004e: Status 404 returned error can't find the container with id 370c3b8f8b496f2d21e7c7280f23424b80657dbbc6bd48a8493caf7f1b11004e
Oct 31 13:24:50.998136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.998110    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92cd891-8b41-4511-9fb0-d724cb1caaf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5\" (UID: \"c92cd891-8b41-4511-9fb0-d724cb1caaf5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"
Oct 31 13:24:50.998279 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.998149    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwp8w\" (UniqueName: \"kubernetes.io/projected/c92cd891-8b41-4511-9fb0-d724cb1caaf5-kube-api-access-dwp8w\") pod \"payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5\" (UID: \"c92cd891-8b41-4511-9fb0-d724cb1caaf5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"
Oct 31 13:24:50.998829 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:50.998807    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92cd891-8b41-4511-9fb0-d724cb1caaf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5\" (UID: \"c92cd891-8b41-4511-9fb0-d724cb1caaf5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"
Oct 31 13:24:51.011666 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:51.011641    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwp8w\" (UniqueName: \"kubernetes.io/projected/c92cd891-8b41-4511-9fb0-d724cb1caaf5-kube-api-access-dwp8w\") pod \"payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5\" (UID: \"c92cd891-8b41-4511-9fb0-d724cb1caaf5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"
Oct 31 13:24:51.029233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:51.029207    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs" event={"ID":"859ad249-88cd-4054-a411-f0847db07b12","Type":"ContainerStarted","Data":"08dc6979326f775489bb6ebcaeb572c9aa2e9aab1dbb3e5fde0e34ed28ab4e9d"}
Oct 31 13:24:51.029332 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:51.029258    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs" event={"ID":"859ad249-88cd-4054-a411-f0847db07b12","Type":"ContainerStarted","Data":"b982a932440f42b347622d9ad123e4cc806e6ad7718d41ae13194f59f1a4888c"}
Oct 31 13:24:51.029391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:51.029367    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"
Oct 31 13:24:51.030430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:51.030408    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp" event={"ID":"7348227d-79bb-4bc8-b6ae-f4e318dcc703","Type":"ContainerStarted","Data":"370c3b8f8b496f2d21e7c7280f23424b80657dbbc6bd48a8493caf7f1b11004e"}
Oct 31 13:24:51.031843 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:51.031812    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w" event={"ID":"b570ecdd-7dda-4021-966d-a36080ef5293","Type":"ContainerStarted","Data":"9aecfa60efd946312185571004b7166b43676f198ea380eab6cc580418e76dd7"}
Oct 31 13:24:51.031965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:51.031882    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w" event={"ID":"b570ecdd-7dda-4021-966d-a36080ef5293","Type":"ContainerStarted","Data":"ebe130c912b8c994155ed78b4b8a9c6573323beb07de02e9ddb648d56872714f"}
Oct 31 13:24:51.051290 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:51.051264    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:51.072249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:51.072201    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs" podStartSLOduration=2.072184181 podStartE2EDuration="2.072184181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:51.049384237 +0000 UTC m=+1288.447732863" watchObservedRunningTime="2026-10-31 13:24:51.072184181 +0000 UTC m=+1288.470532809"
Oct 31 13:24:51.072525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:51.072503    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w" podStartSLOduration=1.072492884 podStartE2EDuration="1.072492884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:51.070977394 +0000 UTC m=+1288.469326019" watchObservedRunningTime="2026-10-31 13:24:51.072492884 +0000 UTC m=+1288.470841508"
Oct 31 13:24:51.080344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:51.080323    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"
Oct 31 13:24:51.359611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:51.359566    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"]
Oct 31 13:24:51.362747 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:24:51.362682    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92cd891_8b41_4511_9fb0_d724cb1caaf5.slice/crio-54e80c625804f599b2a54edd7eff7c1d76779570fe003c216b7202611a1ee2cc WatchSource:0}: Error finding container 54e80c625804f599b2a54edd7eff7c1d76779570fe003c216b7202611a1ee2cc: Status 404 returned error can't find the container with id 54e80c625804f599b2a54edd7eff7c1d76779570fe003c216b7202611a1ee2cc
Oct 31 13:24:52.039064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:52.039028    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp" event={"ID":"7348227d-79bb-4bc8-b6ae-f4e318dcc703","Type":"ContainerStarted","Data":"25bc270a2a63194bfe9a59e2adcef830df0af366088125fd7a6d2dbd36d38c27"}
Oct 31 13:24:52.039245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:52.039105    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"
Oct 31 13:24:52.040749 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:52.040709    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5" event={"ID":"c92cd891-8b41-4511-9fb0-d724cb1caaf5","Type":"ContainerStarted","Data":"ef85f40d2633f3795b91531f2493b1a6f20f80ce6592cbf79dd33e3063e35257"}
Oct 31 13:24:52.040856 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:52.040754    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5" event={"ID":"c92cd891-8b41-4511-9fb0-d724cb1caaf5","Type":"ContainerStarted","Data":"54e80c625804f599b2a54edd7eff7c1d76779570fe003c216b7202611a1ee2cc"}
Oct 31 13:24:52.041100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:52.041084    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"
Oct 31 13:24:52.060526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:52.060501    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:52.061197 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:52.061159    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp" podStartSLOduration=2.061147126 podStartE2EDuration="2.061147126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:52.058284776 +0000 UTC m=+1289.456633401" watchObservedRunningTime="2026-10-31 13:24:52.061147126 +0000 UTC m=+1289.459495749"
Oct 31 13:24:52.080679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:52.080639    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5" podStartSLOduration=2.08062608 podStartE2EDuration="2.08062608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:52.078679721 +0000 UTC m=+1289.477028345" watchObservedRunningTime="2026-10-31 13:24:52.08062608 +0000 UTC m=+1289.478974701"
Oct 31 13:24:52.222100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:52.222047    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:53.051381 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:53.051354    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"
Oct 31 13:24:53.301774 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:53.301688    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:53.457730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:53.457695    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:54.409776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:54.409725    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:55.195770 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:55.195720    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:55.399892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:55.399390    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:55.563894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:55.559236    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:55.998032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:55.997996    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:56.197487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:56.197447    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:56.357573 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:56.357430    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:57.012916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.001924    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"]
Oct 31 13:24:57.016681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.016656    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 13:24:57.038124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.038091    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-8b2f0a-serving-cert\""
Oct 31 13:24:57.038585 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.038550    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 13:24:57.038763 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.038692    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"]
Oct 31 13:24:57.039040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.038894    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bx4t6\""
Oct 31 13:24:57.118782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.099957    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cf0c64-755b-4440-85b0-2657ea896014-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8\" (UID: \"50cf0c64-755b-4440-85b0-2657ea896014\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 13:24:57.162107 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.162078    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55m6v\" (UniqueName: \"kubernetes.io/projected/50cf0c64-755b-4440-85b0-2657ea896014-kube-api-access-55m6v\") pod \"maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8\" (UID: \"50cf0c64-755b-4440-85b0-2657ea896014\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 13:24:57.162423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.162401    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50cf0c64-755b-4440-85b0-2657ea896014-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8\" (UID: \"50cf0c64-755b-4440-85b0-2657ea896014\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 13:24:57.162665 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.162643    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50cf0c64-755b-4440-85b0-2657ea896014-metrics-certs\") pod \"maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8\" (UID: \"50cf0c64-755b-4440-85b0-2657ea896014\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 13:24:57.185999 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.179722    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:57.281950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.281531    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cf0c64-755b-4440-85b0-2657ea896014-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8\" (UID: \"50cf0c64-755b-4440-85b0-2657ea896014\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 13:24:57.281950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.281584    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55m6v\" (UniqueName: \"kubernetes.io/projected/50cf0c64-755b-4440-85b0-2657ea896014-kube-api-access-55m6v\") pod \"maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8\" (UID: \"50cf0c64-755b-4440-85b0-2657ea896014\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 13:24:57.281950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.281615    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50cf0c64-755b-4440-85b0-2657ea896014-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8\" (UID: \"50cf0c64-755b-4440-85b0-2657ea896014\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 13:24:57.281950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.281673    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50cf0c64-755b-4440-85b0-2657ea896014-metrics-certs\") pod \"maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8\" (UID: \"50cf0c64-755b-4440-85b0-2657ea896014\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 13:24:57.283210 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.283168    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cf0c64-755b-4440-85b0-2657ea896014-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8\" (UID: \"50cf0c64-755b-4440-85b0-2657ea896014\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 13:24:57.302883 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.302791    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50cf0c64-755b-4440-85b0-2657ea896014-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8\" (UID: \"50cf0c64-755b-4440-85b0-2657ea896014\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 13:24:57.303731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.303711    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50cf0c64-755b-4440-85b0-2657ea896014-metrics-certs\") pod \"maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8\" (UID: \"50cf0c64-755b-4440-85b0-2657ea896014\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 13:24:57.327677 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.327638    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55m6v\" (UniqueName: \"kubernetes.io/projected/50cf0c64-755b-4440-85b0-2657ea896014-kube-api-access-55m6v\") pod \"maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8\" (UID: \"50cf0c64-755b-4440-85b0-2657ea896014\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 13:24:57.342743 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.342660    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:57.461892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.461061    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 13:24:57.967888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:57.967733    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:58.324894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:58.296083    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"]
Oct 31 13:24:58.799458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:58.799394    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:24:59.361066 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:59.361025    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8" event={"ID":"50cf0c64-755b-4440-85b0-2657ea896014","Type":"ContainerStarted","Data":"dcb0fbde9ad49fc445bf9090813fda2732c3bb82894195c9ad593d6f478f6fdb"}
Oct 31 13:24:59.851892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:24:59.788210    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:00.030183 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:00.030100    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"
Oct 31 13:25:00.863983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:00.863858    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:01.034973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:01.034944    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"
Oct 31 13:25:01.042999 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:01.042969    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:25:01.043128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:01.043011    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj" podUID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:25:01.043991 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:01.043972    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:25:01.044073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:01.044004    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:25:01.370455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:01.370340    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:01.531605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:01.531560    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:01.887129 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:01.886400    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:02.076063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:02.075060    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"
Oct 31 13:25:02.077677 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:02.077015    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"
Oct 31 13:25:02.104499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:02.101355    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:02.304610 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:02.304522    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:02.500715 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:02.500596    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:02.700947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:02.694552    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:03.087898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:03.085837    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"
Oct 31 13:25:03.089682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:03.088551    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"
Oct 31 13:25:03.105957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:03.105548    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:03.395367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:03.395332    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:03.630646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:03.630533    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:04.073826 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:04.073737    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"
Oct 31 13:25:04.192232 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:04.192190    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:04.455777 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:04.455687    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8" event={"ID":"50cf0c64-755b-4440-85b0-2657ea896014","Type":"ContainerStarted","Data":"9adaacb7fac9c5a70aba8bd74af5132a7c9044fc2c14e0abbd6337258172550f"}
Oct 31 13:25:04.456702 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:04.456682    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 13:25:04.490362 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:04.490292    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8" podStartSLOduration=3.451310421 podStartE2EDuration="8.490272748s" totalImagesPullingTime="5.038962327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:04.487962616 +0000 UTC m=+1301.886311241" watchObservedRunningTime="2026-10-31 13:25:04.490272748 +0000 UTC m=+1301.888621372"
Oct 31 13:25:05.059676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:05.059634    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:06.278286 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:06.278244    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:07.285393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:07.285351    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:07.443015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:07.442982    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:07.909431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:07.909395    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:08.099569 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:08.079248    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:08.262442 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:08.262410    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:08.541418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:08.489270    4428 generic.go:356] "Generic (PLEG): container finished" podID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerID="71bc7b5aabe8ff9a363889f5436413ba9693154a01dcd88ae0d9437b88bf20f4" exitCode=0
Oct 31 13:25:08.541418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:08.489335    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" event={"ID":"1667cc7b-35eb-4c92-9fa7-239aaa5990d1","Type":"ContainerDied","Data":"71bc7b5aabe8ff9a363889f5436413ba9693154a01dcd88ae0d9437b88bf20f4"}
Oct 31 13:25:08.541418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:08.489698    4428 scope.go:122] "RemoveContainer" containerID="71bc7b5aabe8ff9a363889f5436413ba9693154a01dcd88ae0d9437b88bf20f4"
Oct 31 13:25:08.541418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:08.491175    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:25:08.541418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:08.491249    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:25:08.541418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:08.491295    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:25:08.546329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:08.546300    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:09.208102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.208061    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:09.500498 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.500456    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" event={"ID":"1667cc7b-35eb-4c92-9fa7-239aaa5990d1","Type":"ContainerStarted","Data":"de4ab4fad18c7d488c8dc5b9464b2c326ab943da71609bd6be99190bdeca3980"}
Oct 31 13:25:09.502091 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.502066    4428 generic.go:356] "Generic (PLEG): container finished" podID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" containerID="4bc53ab41306a1cd1c8e0ce93d07ffc45d20dcc0f3d3676ef4e3185c5fcfbe05" exitCode=0
Oct 31 13:25:09.502233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.502127    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj" event={"ID":"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6","Type":"ContainerDied","Data":"4bc53ab41306a1cd1c8e0ce93d07ffc45d20dcc0f3d3676ef4e3185c5fcfbe05"}
Oct 31 13:25:09.502489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.502467    4428 scope.go:122] "RemoveContainer" containerID="4bc53ab41306a1cd1c8e0ce93d07ffc45d20dcc0f3d3676ef4e3185c5fcfbe05"
Oct 31 13:25:09.503555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.503529    4428 generic.go:356] "Generic (PLEG): container finished" podID="d5e6e17a-543e-4373-ad93-e0ed60c74334" containerID="f4c42ef64539004b74cd50af897ac2cedec54f5db0d8bb70a148211da1ad9041" exitCode=0
Oct 31 13:25:09.503636 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.503599    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj" event={"ID":"d5e6e17a-543e-4373-ad93-e0ed60c74334","Type":"ContainerDied","Data":"f4c42ef64539004b74cd50af897ac2cedec54f5db0d8bb70a148211da1ad9041"}
Oct 31 13:25:09.503783 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.503756    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:25:09.503859 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.503820    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:25:09.503859 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.503851    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:25:09.503965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.503909    4428 scope.go:122] "RemoveContainer" containerID="f4c42ef64539004b74cd50af897ac2cedec54f5db0d8bb70a148211da1ad9041"
Oct 31 13:25:09.505194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.505163    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:25:09.505278 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.505236    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:25:09.505334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.505279    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:25:09.505772 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.505751    4428 generic.go:356] "Generic (PLEG): container finished" podID="6002eb05-9237-48f2-972a-448248703ca4" containerID="86f4b0c34be4808cb41009e58c6fb3a9ddd7aab94ae09bd4c85399f1e7fa1580" exitCode=0
Oct 31 13:25:09.505890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.505845    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld" event={"ID":"6002eb05-9237-48f2-972a-448248703ca4","Type":"ContainerDied","Data":"86f4b0c34be4808cb41009e58c6fb3a9ddd7aab94ae09bd4c85399f1e7fa1580"}
Oct 31 13:25:09.506280 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.506212    4428 scope.go:122] "RemoveContainer" containerID="86f4b0c34be4808cb41009e58c6fb3a9ddd7aab94ae09bd4c85399f1e7fa1580"
Oct 31 13:25:09.507408 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.507381    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:25:09.507474 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.507457    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:25:09.507536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:09.507506    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:25:10.100507 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:10.100420    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:10.227235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:10.224434    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:25:10.231892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:10.228287    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 13:25:10.231892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:10.228353    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 13:25:10.593808 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:10.593773    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld" event={"ID":"6002eb05-9237-48f2-972a-448248703ca4","Type":"ContainerStarted","Data":"f6d17cdd09c7023eefe9f5e0d29eba4024fa3123d0ad91241fea7b3575f4dd03"}
Oct 31 13:25:10.675045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:10.675010    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj" event={"ID":"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6","Type":"ContainerStarted","Data":"3b4362c55cd284b08313d0411fdca699da3a8f057ced98851c18b17bbaf45d1f"}
Oct 31 13:25:10.695889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:10.692107    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj" event={"ID":"d5e6e17a-543e-4373-ad93-e0ed60c74334","Type":"ContainerStarted","Data":"b2c4e91a017758f0511a49cac163f50e1fac4d2a41a2b8fbe9ae97f034bc7a7f"}
Oct 31 13:25:10.795858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:10.795826    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 31 13:25:10.807698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:10.807655    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld" podUID="6002eb05-9237-48f2-972a-448248703ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 31 13:25:10.808218 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:10.798825    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:25:10.983904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:10.983084    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:11.223894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:11.223262    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 13:25:11.223894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:11.223338    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 13:25:11.477961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:11.458288    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:25:11.477961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:11.458463    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 13:25:11.477961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:11.458508    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj" podUID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 13:25:11.510895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:11.510025    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 13:25:11.717900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:11.673068    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:25:11.751944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:11.722997    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 31 13:25:11.751944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:11.723062    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj" podUID="d5e6e17a-543e-4373-ad93-e0ed60c74334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 31 13:25:11.814895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:11.784447    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 31 13:25:11.814895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:11.784509    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld" podUID="6002eb05-9237-48f2-972a-448248703ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 31 13:25:11.852896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:11.852098    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:12.020947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.017927    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"]
Oct 31 13:25:12.073109 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.073068    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"]
Oct 31 13:25:12.073311 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.073203    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.153898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.152483    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5286ea\"/\"kube-root-ca.crt\""
Oct 31 13:25:12.153898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.153587    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5286ea\"/\"simulator-w2-5286ea-kserve-self-signed-certs\""
Oct 31 13:25:12.153898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.153684    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5286ea\"/\"default-dockercfg-r7fxl\""
Oct 31 13:25:12.153898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.153800    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5286ea\"/\"openshift-service-ca.crt\""
Oct 31 13:25:12.172782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.156437    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lkh5\" (UniqueName: \"kubernetes.io/projected/ab6906eb-42bc-426f-9861-4e5af42cc54d-kube-api-access-5lkh5\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.172782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.156493    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-tmp-dir\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.172782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.156553    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-home\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.172782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.156604    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-model-cache\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.172782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.156627    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab6906eb-42bc-426f-9861-4e5af42cc54d-tls-certs\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.172782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.156712    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-dshm\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.304948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.301338    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:12.384434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.350531    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-dshm\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.384434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.350604    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lkh5\" (UniqueName: \"kubernetes.io/projected/ab6906eb-42bc-426f-9861-4e5af42cc54d-kube-api-access-5lkh5\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.384434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.350661    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-tmp-dir\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.384434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.350738    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-home\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.384434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.350801    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-model-cache\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.384434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.350831    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab6906eb-42bc-426f-9861-4e5af42cc54d-tls-certs\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.400150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.400121    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-home\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.400813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.400791    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-model-cache\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.401176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.401154    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 13:25:12.401339 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.401314    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 13:25:12.458129 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.404771    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-dshm\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.479195 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.467235    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-tmp-dir\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.501261 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.501229    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 13:25:12.501882 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.501835    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj" podUID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 13:25:12.502549 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.502524    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab6906eb-42bc-426f-9861-4e5af42cc54d-tls-certs\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.504678 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.504653    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:12.551896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.545963    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lkh5\" (UniqueName: \"kubernetes.io/projected/ab6906eb-42bc-426f-9861-4e5af42cc54d-kube-api-access-5lkh5\") pod \"simulator-w2-5286ea-kserve-6455577db4-w6gf4\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") " pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.668306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.668217    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 31 13:25:12.669942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.669860    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj" podUID="d5e6e17a-543e-4373-ad93-e0ed60c74334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 31 13:25:12.731106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.731072    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:12.807650 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.807616    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 31 13:25:12.836817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.824399    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:12.837643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:12.837607    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld" podUID="6002eb05-9237-48f2-972a-448248703ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 31 13:25:13.032771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:13.032737    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:13.225842 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:13.225810    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:13.230486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:13.230455    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 13:25:13.230699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:13.230678    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 13:25:13.738920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:13.728627    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 13:25:13.738920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:13.728683    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj" podUID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 13:25:13.738920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:13.729764    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 31 13:25:13.738920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:13.729806    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj" podUID="d5e6e17a-543e-4373-ad93-e0ed60c74334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 31 13:25:13.757896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:13.745723    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 31 13:25:13.757896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:13.745768    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld" podUID="6002eb05-9237-48f2-972a-448248703ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 31 13:25:13.894489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:13.878236    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"]
Oct 31 13:25:13.956893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:13.931933    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:14.144136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:14.141372    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:14.288893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:14.276311    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 13:25:14.288893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:14.276369    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 13:25:14.469895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:14.469454    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 13:25:14.469895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:14.469531    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj" podUID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 13:25:14.671045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:14.670956    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 31 13:25:14.671881 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:14.671833    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj" podUID="d5e6e17a-543e-4373-ad93-e0ed60c74334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 31 13:25:14.748306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:14.748267    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 31 13:25:14.748918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:14.748854    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld" podUID="6002eb05-9237-48f2-972a-448248703ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 31 13:25:14.800005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:14.799972    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:14.953069 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:14.952977    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:25:14.954819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:14.954796    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:25:15.061324 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:15.061280    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" event={"ID":"ab6906eb-42bc-426f-9861-4e5af42cc54d","Type":"ContainerStarted","Data":"e630cb9ca71b25fdf7e82475339ac1b6d7f3d9ca4b1fb0cbd4af7f8a9796b66b"}
Oct 31 13:25:15.061593 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:15.061571    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" event={"ID":"ab6906eb-42bc-426f-9861-4e5af42cc54d","Type":"ContainerStarted","Data":"a61f59d32064f4623391fdea4b723148176a8e951f3e1f7fc845779d95ccee82"}
Oct 31 13:25:15.092900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:15.084611    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podStartSLOduration=4.084592316 podStartE2EDuration="4.084592316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:15.083429101 +0000 UTC m=+1312.481777731" watchObservedRunningTime="2026-10-31 13:25:15.084592316 +0000 UTC m=+1312.482940941"
Oct 31 13:25:15.229056 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:15.229016    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 13:25:15.229354 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:15.229319    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 13:25:15.458679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:15.458242    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 13:25:15.458679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:15.458301    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj" podUID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 13:25:15.613800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:15.613718    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:15.700224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:15.700192    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 31 13:25:15.700495 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:15.700465    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj" podUID="d5e6e17a-543e-4373-ad93-e0ed60c74334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 31 13:25:15.734999 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:15.734943    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:15.752804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:15.752373    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 31 13:25:15.752804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:15.752427    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld" podUID="6002eb05-9237-48f2-972a-448248703ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 31 13:25:15.964951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:15.962970    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:25:15.964951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:15.963947    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:25:16.224464 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:16.224371    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 13:25:16.224710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:16.224681    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 13:25:16.317789 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:16.317717    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:16.485081 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:16.484984    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 13:25:16.485954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:16.485920    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj" podUID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 13:25:16.671254 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:16.671217    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 31 13:25:16.671539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:16.671511    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj" podUID="d5e6e17a-543e-4373-ad93-e0ed60c74334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 31 13:25:16.754068 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:16.753967    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 31 13:25:16.754716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:16.754681    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld" podUID="6002eb05-9237-48f2-972a-448248703ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 31 13:25:17.224953 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.224922    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:25:17.226185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.226156    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:25:17.336089 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.336025    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:17.411134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.411098    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"]
Oct 31 13:25:17.432057 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.432022    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"]
Oct 31 13:25:17.432287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.432144    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.441557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.440564    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7b3ddd\"/\"simulator-w0-7b3ddd-kserve-self-signed-certs\""
Oct 31 13:25:17.441557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.441268    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7b3ddd\"/\"kube-root-ca.crt\""
Oct 31 13:25:17.441557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.441356    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7b3ddd\"/\"openshift-service-ca.crt\""
Oct 31 13:25:17.441557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.441493    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7b3ddd\"/\"default-dockercfg-bqk6t\""
Oct 31 13:25:17.532359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.532257    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2ff4c9-6250-41b6-b780-e5f189139e59-tls-certs\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.532359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.532307    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-home\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.532359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.532326    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-tmp-dir\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.532638 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.532511    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-dshm\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.532638 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.532582    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gmkl\" (UniqueName: \"kubernetes.io/projected/4f2ff4c9-6250-41b6-b780-e5f189139e59-kube-api-access-5gmkl\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.532750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.532644    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-model-cache\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.563017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.562972    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:17.633403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.633342    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gmkl\" (UniqueName: \"kubernetes.io/projected/4f2ff4c9-6250-41b6-b780-e5f189139e59-kube-api-access-5gmkl\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.633622 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.633454    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-model-cache\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.633622 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.633494    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2ff4c9-6250-41b6-b780-e5f189139e59-tls-certs\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.633622 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.633525    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-home\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.633622 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.633553    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-tmp-dir\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.633894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.633625    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-dshm\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.634104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.634057    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-home\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.634228 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.634155    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-model-cache\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.634518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.634482    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-tmp-dir\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.638468 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.638446    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2ff4c9-6250-41b6-b780-e5f189139e59-tls-certs\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.644128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.644105    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-dshm\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.647679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.647655    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gmkl\" (UniqueName: \"kubernetes.io/projected/4f2ff4c9-6250-41b6-b780-e5f189139e59-kube-api-access-5gmkl\") pod \"simulator-w0-7b3ddd-kserve-56d787bdc7-hx642\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") " pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:17.658539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.658518    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:25:17.661486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.661428    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 13:25:17.774492 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.774453    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:17.791247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:17.791153    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:18.024630 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:18.024588    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:25:18.045391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:18.045316    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:25:18.110188 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:18.107263    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"]
Oct 31 13:25:18.110188 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:25:18.110023    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f2ff4c9_6250_41b6_b780_e5f189139e59.slice/crio-1707362bd6b181b2ea9f951f10a6b420291035276b9a7443569d901ce1d40f90 WatchSource:0}: Error finding container 1707362bd6b181b2ea9f951f10a6b420291035276b9a7443569d901ce1d40f90: Status 404 returned error can't find the container with id 1707362bd6b181b2ea9f951f10a6b420291035276b9a7443569d901ce1d40f90
Oct 31 13:25:18.213141 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:18.212569    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" event={"ID":"4f2ff4c9-6250-41b6-b780-e5f189139e59","Type":"ContainerStarted","Data":"3f3782d8b48f18b56dd672c86e1a1b6fa8b9193efb564bd1d12664266ae18012"}
Oct 31 13:25:18.213141 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:18.212619    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" event={"ID":"4f2ff4c9-6250-41b6-b780-e5f189139e59","Type":"ContainerStarted","Data":"1707362bd6b181b2ea9f951f10a6b420291035276b9a7443569d901ce1d40f90"}
Oct 31 13:25:18.233762 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:18.233719    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podStartSLOduration=1.233702166 podStartE2EDuration="1.233702166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:18.232918448 +0000 UTC m=+1315.631267072" watchObservedRunningTime="2026-10-31 13:25:18.233702166 +0000 UTC m=+1315.632050791"
Oct 31 13:25:18.387463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:18.387386    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:25:18.388585 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:18.388563    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 13:25:18.743436 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:25:18.743398    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:25:18.792175 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:18.792140    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:19.218176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:19.218148    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:19.372975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:19.372943    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:19.998607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:19.998379    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:20.811483 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:20.811451    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:21.511939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:21.511842    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:21.665074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:21.665045    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:22.395108 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:22.395076    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:22.731675 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:22.731650    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:22.744368 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:22.744350    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:23.217928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:23.217891    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:23.250419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:23.250396    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:25:23.996526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:23.996478    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:24.924893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:24.922853    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:25.797674 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:25.797633    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:26.592280 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:26.592218    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:27.441972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:27.441938    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:27.789365 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:27.789332    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:27.791575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:27.791553    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:27.803922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:27.803904    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:28.276591 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:28.276557    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:25:28.365112 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:28.365082    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:29.299009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:29.298970    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:30.070681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:30.070646    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:30.973986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:30.970663    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:31.710836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:31.710792    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:32.399523 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:32.399488    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:32.562789 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:32.562758    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:33.506186 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:33.506151    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:34.324538 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:34.324502    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:35.121053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:35.120912    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:35.986938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:35.986903    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:36.812462 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:36.812348    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:37.624450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:37.624417    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:38.417912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:38.417859    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:39.196895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.193975    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:39.654738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.654683    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"]
Oct 31 13:25:39.669546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.669517    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.671556 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.671531    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"]
Oct 31 13:25:39.675884 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.675830    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c82140\"/\"openshift-service-ca.crt\""
Oct 31 13:25:39.676002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.675930    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c82140\"/\"kube-root-ca.crt\""
Oct 31 13:25:39.676145 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.676131    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c82140\"/\"premium-w3-c82140-kserve-self-signed-certs\""
Oct 31 13:25:39.676236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.676183    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c82140\"/\"default-dockercfg-mzt5g\""
Oct 31 13:25:39.770413 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.770383    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:39.835134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.834632    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-dshm\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.835134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.834734    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pvjg\" (UniqueName: \"kubernetes.io/projected/1101d3a2-50ce-47fb-913c-4f40194fcdf1-kube-api-access-2pvjg\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.835134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.834773    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-tmp-dir\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.835134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.834815    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-model-cache\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.835134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.834950    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-home\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.835134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.834977    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1101d3a2-50ce-47fb-913c-4f40194fcdf1-tls-certs\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.935725 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.935646    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-model-cache\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.935928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.935725    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-home\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.935928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.935746    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1101d3a2-50ce-47fb-913c-4f40194fcdf1-tls-certs\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.935928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.935779    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-dshm\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.935928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.935825    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pvjg\" (UniqueName: \"kubernetes.io/projected/1101d3a2-50ce-47fb-913c-4f40194fcdf1-kube-api-access-2pvjg\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.935928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.935851    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-tmp-dir\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.936233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.936090    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-model-cache\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.936233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.936215    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-home\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.936317 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.936295    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-tmp-dir\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.938239 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.938214    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-dshm\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.938521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.938505    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1101d3a2-50ce-47fb-913c-4f40194fcdf1-tls-certs\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.949323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.949298    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pvjg\" (UniqueName: \"kubernetes.io/projected/1101d3a2-50ce-47fb-913c-4f40194fcdf1-kube-api-access-2pvjg\") pod \"premium-w3-c82140-kserve-6958b88b5-xxshv\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") " pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:39.985360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:39.985336    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:40.032099 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:40.032044    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:40.294001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:40.293980    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"]
Oct 31 13:25:40.295710 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:25:40.295683    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1101d3a2_50ce_47fb_913c_4f40194fcdf1.slice/crio-eb1dd874a87dc5b54d87cc701106d819deeaa639eedc1ec23b71fae9790fd368 WatchSource:0}: Error finding container eb1dd874a87dc5b54d87cc701106d819deeaa639eedc1ec23b71fae9790fd368: Status 404 returned error can't find the container with id eb1dd874a87dc5b54d87cc701106d819deeaa639eedc1ec23b71fae9790fd368
Oct 31 13:25:40.353264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:40.353232    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" event={"ID":"1101d3a2-50ce-47fb-913c-4f40194fcdf1","Type":"ContainerStarted","Data":"eb1dd874a87dc5b54d87cc701106d819deeaa639eedc1ec23b71fae9790fd368"}
Oct 31 13:25:40.893372 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:40.893324    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:41.359981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:41.359940    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" event={"ID":"1101d3a2-50ce-47fb-913c-4f40194fcdf1","Type":"ContainerStarted","Data":"8481efb45d6fcdae9ac3557513c75466e9b7bb889bbd620f45d3d6206dadddab"}
Oct 31 13:25:41.380166 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:41.380119    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podStartSLOduration=2.380106074 podStartE2EDuration="2.380106074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:41.379384765 +0000 UTC m=+1338.777733391" watchObservedRunningTime="2026-10-31 13:25:41.380106074 +0000 UTC m=+1338.778454698"
Oct 31 13:25:41.684390 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:41.682679    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:41.985440 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:41.985413    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:42.497352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:42.497318    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:43.289504 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:43.286355    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:44.150146 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.150116    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"]
Oct 31 13:25:44.154047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.154028    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.162827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.162804    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5286ea\"/\"premium-w2-5286ea-kserve-self-signed-certs\""
Oct 31 13:25:44.165565 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.165544    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"]
Oct 31 13:25:44.167634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.167605    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:44.209202 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.209172    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6lwm\" (UniqueName: \"kubernetes.io/projected/beb4582b-cb4b-437b-b527-6e77941a2543-kube-api-access-w6lwm\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.209317 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.209218    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-home\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.209317 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.209234    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-dshm\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.209397 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.209362    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-tmp-dir\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.209397 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.209387    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-model-cache\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.209487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.209471    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb4582b-cb4b-437b-b527-6e77941a2543-tls-certs\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.313736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.313687    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6lwm\" (UniqueName: \"kubernetes.io/projected/beb4582b-cb4b-437b-b527-6e77941a2543-kube-api-access-w6lwm\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.314786 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.314757    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-home\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.315526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.315510    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-dshm\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.319047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.316129    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-tmp-dir\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.319560 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.319534    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-model-cache\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.320801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.320077    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb4582b-cb4b-437b-b527-6e77941a2543-tls-certs\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.321467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.315444    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-home\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.322101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.319961    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-model-cache\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.322101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.319469    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-tmp-dir\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.332887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.328236    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-dshm\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.332887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.328774    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb4582b-cb4b-437b-b527-6e77941a2543-tls-certs\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.332887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.330591    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6lwm\" (UniqueName: \"kubernetes.io/projected/beb4582b-cb4b-437b-b527-6e77941a2543-kube-api-access-w6lwm\") pod \"premium-w2-5286ea-kserve-649fcb656f-gr7p5\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") " pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.367381 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.367342    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:44.468766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.468735    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:44.769555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:44.769528    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"]
Oct 31 13:25:44.771733 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:25:44.771704    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbeb4582b_cb4b_437b_b527_6e77941a2543.slice/crio-cf981b441427c13bf8ebed73424a31d187e74c0dd95ea09386e2668053c072e2 WatchSource:0}: Error finding container cf981b441427c13bf8ebed73424a31d187e74c0dd95ea09386e2668053c072e2: Status 404 returned error can't find the container with id cf981b441427c13bf8ebed73424a31d187e74c0dd95ea09386e2668053c072e2
Oct 31 13:25:45.114236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:45.111804    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:45.386113 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:45.386042    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" event={"ID":"beb4582b-cb4b-437b-b527-6e77941a2543","Type":"ContainerStarted","Data":"9dcfcf33b0a714bb03461ae1a28a0f14a6aa93649426c6e037567bb91e2a913d"}
Oct 31 13:25:45.386113 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:45.386078    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" event={"ID":"beb4582b-cb4b-437b-b527-6e77941a2543","Type":"ContainerStarted","Data":"cf981b441427c13bf8ebed73424a31d187e74c0dd95ea09386e2668053c072e2"}
Oct 31 13:25:45.418043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:45.417990    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podStartSLOduration=1.417975392 podStartE2EDuration="1.417975392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:45.417095883 +0000 UTC m=+1342.815444538" watchObservedRunningTime="2026-10-31 13:25:45.417975392 +0000 UTC m=+1342.816324015"
Oct 31 13:25:45.469697 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:45.469672    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:46.103575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:46.103525    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:46.990650 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:46.990611    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:47.179385 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:47.179342    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:48.022147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.022111    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:48.613157 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.613123    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"]
Oct 31 13:25:48.617419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.617399    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.637511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.637486    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aabeef\"/\"premium-w6-aabeef-kserve-self-signed-certs\""
Oct 31 13:25:48.637660 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.637559    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-aabeef\"/\"openshift-service-ca.crt\""
Oct 31 13:25:48.637660 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.637616    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-aabeef\"/\"kube-root-ca.crt\""
Oct 31 13:25:48.640821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.640793    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"]
Oct 31 13:25:48.642085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.642066    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aabeef\"/\"default-dockercfg-9t4bh\""
Oct 31 13:25:48.685924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.685897    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6a586d-e1a2-4965-b0fc-89e2286d7298-tls-certs\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.686080 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.685935    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-dshm\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.686080 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.685955    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-model-cache\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.686080 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.685973    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-home\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.686180 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.686148    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-tmp-dir\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.686216 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.686192    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llrqj\" (UniqueName: \"kubernetes.io/projected/5d6a586d-e1a2-4965-b0fc-89e2286d7298-kube-api-access-llrqj\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.736302 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.736269    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:48.741022 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:25:48.740987    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:25:48.787360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.787332    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-tmp-dir\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.787526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.787377    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llrqj\" (UniqueName: \"kubernetes.io/projected/5d6a586d-e1a2-4965-b0fc-89e2286d7298-kube-api-access-llrqj\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.787526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.787460    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6a586d-e1a2-4965-b0fc-89e2286d7298-tls-certs\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.787526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.787482    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-dshm\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.787526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.787501    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-model-cache\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.787742 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.787522    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-home\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.787801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.787774    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-tmp-dir\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.787919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.787896    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-home\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.787978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.787961    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-model-cache\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.789842 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.789820    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-dshm\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.790084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.790067    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6a586d-e1a2-4965-b0fc-89e2286d7298-tls-certs\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.800238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.800215    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llrqj\" (UniqueName: \"kubernetes.io/projected/5d6a586d-e1a2-4965-b0fc-89e2286d7298-kube-api-access-llrqj\") pod \"premium-w6-aabeef-kserve-575d86888d-7hvjn\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") " pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:48.899426 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.899331    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:48.929802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:48.929778    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:49.216842 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:49.216814    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"]
Oct 31 13:25:49.220541 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:25:49.220510    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d6a586d_e1a2_4965_b0fc_89e2286d7298.slice/crio-7f9446ccf3fad8d35639e7b591e24955d0e85c803f332e069231181050f7bd25 WatchSource:0}: Error finding container 7f9446ccf3fad8d35639e7b591e24955d0e85c803f332e069231181050f7bd25: Status 404 returned error can't find the container with id 7f9446ccf3fad8d35639e7b591e24955d0e85c803f332e069231181050f7bd25
Oct 31 13:25:49.409311 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:49.409270    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" event={"ID":"5d6a586d-e1a2-4965-b0fc-89e2286d7298","Type":"ContainerStarted","Data":"deab884524b3c7337469c5258a4440cf3037e44583dddc6ac06da2185285c568"}
Oct 31 13:25:49.409311 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:49.409320    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" event={"ID":"5d6a586d-e1a2-4965-b0fc-89e2286d7298","Type":"ContainerStarted","Data":"7f9446ccf3fad8d35639e7b591e24955d0e85c803f332e069231181050f7bd25"}
Oct 31 13:25:49.453746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:49.453692    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podStartSLOduration=1.453677249 podStartE2EDuration="1.453677249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:49.449999765 +0000 UTC m=+1346.848348391" watchObservedRunningTime="2026-10-31 13:25:49.453677249 +0000 UTC m=+1346.852025873"
Oct 31 13:25:49.633203 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:49.631842    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:49.930034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:49.929959    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:49.986429 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:49.986403    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:49.998662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:49.998641    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:50.426889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:50.426848    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:25:50.534363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:50.534332    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:51.639191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:51.639158    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:52.273625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:52.273590    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:52.517152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:52.517087    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:53.410892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:53.410277    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:54.306287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:54.306233    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:54.468947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:54.468895    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:54.488635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:54.488613    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:55.309110 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:55.309078    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:55.455374 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:55.455346    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:25:55.923765 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:55.923733    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:56.336224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:56.336191    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:57.104644 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:57.104592    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:57.853619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:57.853561    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:58.014262 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:58.014193    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:58.930732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:58.930687    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:58.932731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:58.932701    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:25:58.943502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:58.943483    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:59.476139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:59.476108    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:25:59.801836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:25:59.801747    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:00.597661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:00.597628    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:01.436204 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:01.436159    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:02.251501 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:02.251470    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:03.014854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:03.014822    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:03.799619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:03.799587    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:04.600746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:04.600712    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:05.410808 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:05.410778    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:06.423478 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:06.423402    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:07.615077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:07.615041    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:08.851020 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:08.850987    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:09.835472 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:09.835438    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:10.770191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:10.770165    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:11.472261 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:11.472219    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:12.242814 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:12.242760    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:12.404063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:12.404011    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:13.202211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:13.202176    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:14.111293 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:14.111261    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:14.928490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:14.928442    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:15.788768 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:15.788668    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:16.667259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:16.667221    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:17.848895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:17.845989    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:18.739324 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:26:18.739290    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:26:18.984424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:18.984395    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:19.932766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:19.932731    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:20.918263 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:20.918225    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:21.955243 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:21.955205    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:23.105738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:23.105323    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:23.922357 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:23.922311    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:24.824036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:24.823982    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:25.611496 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:25.611458    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:26.422390 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:26.422341    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:27.105557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:27.105490    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:27.823827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:27.823788    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:28.209178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.209099    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"]
Oct 31 13:26:28.213288 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.213265    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.230399 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.230375    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"]
Oct 31 13:26:28.239794 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.239770    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7b3ddd-openshift-default-dockercfg-lzzgx\""
Oct 31 13:26:28.281775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.281753    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.281912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.281793    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.281912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.281813    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208fddf2-00ef-4f4a-944a-f3d93ba37911-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.282000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.281922    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.282000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.281957    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.282000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.281972    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.282140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.282002    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb89t\" (UniqueName: \"kubernetes.io/projected/208fddf2-00ef-4f4a-944a-f3d93ba37911-kube-api-access-fb89t\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.282140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.282025    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.282218 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.282166    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.383101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.383070    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.383234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.383117    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.383234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.383149    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208fddf2-00ef-4f4a-944a-f3d93ba37911-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.383234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.383186    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.383465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.383236    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.383465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.383262    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.383465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.383306    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb89t\" (UniqueName: \"kubernetes.io/projected/208fddf2-00ef-4f4a-944a-f3d93ba37911-kube-api-access-fb89t\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.383465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.383347    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.383465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.383406    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.383724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.383473    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.383724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.383504    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.383724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.383617    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.383900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.383800    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.384101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.384083    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208fddf2-00ef-4f4a-944a-f3d93ba37911-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.385787 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.385765    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.385965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.385949    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.397877 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.397841    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.397969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.397861    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb89t\" (UniqueName: \"kubernetes.io/projected/208fddf2-00ef-4f4a-944a-f3d93ba37911-kube-api-access-fb89t\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.434660 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.434634    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"]
Oct 31 13:26:28.440182 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.440161    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.445942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.445924    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8b2f0a-openshift-default-dockercfg-w6shk\""
Oct 31 13:26:28.454143 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.454119    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"]
Oct 31 13:26:28.527518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.527491    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:28.585736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.585706    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.586340 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.585942    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c64ba1-f422-427b-80b8-1f464d32fafe-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.586740 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.586716    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.587007 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.586988    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.587152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.587134    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.587274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.587260    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.587401 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.587385    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.587619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.587604    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bfs5\" (UniqueName: \"kubernetes.io/projected/a7c64ba1-f422-427b-80b8-1f464d32fafe-kube-api-access-7bfs5\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.587744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.587724    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.688565 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.688526    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.688565 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.688566    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c64ba1-f422-427b-80b8-1f464d32fafe-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.688803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.688610    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.688803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.688641    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.688803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.688672    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.688803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.688704    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.688803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.688740    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.689097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.688838    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bfs5\" (UniqueName: \"kubernetes.io/projected/a7c64ba1-f422-427b-80b8-1f464d32fafe-kube-api-access-7bfs5\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.689097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.688934    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.689097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.689066    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.689322 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.689299    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.689586 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.689564    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.689688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.689598    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.689918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.689894    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c64ba1-f422-427b-80b8-1f464d32fafe-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.691114 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.691096    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.691465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.691450    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.701824 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.701798    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.701927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.701879    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bfs5\" (UniqueName: \"kubernetes.io/projected/a7c64ba1-f422-427b-80b8-1f464d32fafe-kube-api-access-7bfs5\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.758143 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.757716    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:28.768341 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.768301    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:28.821862 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.820645    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"]
Oct 31 13:26:28.824904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.824760    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:26:28.824904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.824837    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:26:28.825370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:28.824988    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:26:29.064718 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:26:29.064688    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c64ba1_f422_427b_80b8_1f464d32fafe.slice/crio-669145f1fca1dcf30168f058f873faa47ad1120b52aa0520bf74a08e20b60809 WatchSource:0}: Error finding container 669145f1fca1dcf30168f058f873faa47ad1120b52aa0520bf74a08e20b60809: Status 404 returned error can't find the container with id 669145f1fca1dcf30168f058f873faa47ad1120b52aa0520bf74a08e20b60809
Oct 31 13:26:29.064910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:29.064771    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"]
Oct 31 13:26:29.067154 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:29.067120    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:26:29.067271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:29.067185    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:26:29.067271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:29.067213    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:26:29.654894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:29.651727    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:29.658104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:29.658067    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h" event={"ID":"a7c64ba1-f422-427b-80b8-1f464d32fafe","Type":"ContainerStarted","Data":"e5971e16f8ebf07d4b0aa829efb4b5d4c3f4e6520f11802d7cfdc36d1ef4012a"}
Oct 31 13:26:29.658373 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:29.658346    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h" event={"ID":"a7c64ba1-f422-427b-80b8-1f464d32fafe","Type":"ContainerStarted","Data":"669145f1fca1dcf30168f058f873faa47ad1120b52aa0520bf74a08e20b60809"}
Oct 31 13:26:29.688524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:29.687684    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj" event={"ID":"208fddf2-00ef-4f4a-944a-f3d93ba37911","Type":"ContainerStarted","Data":"bf98b31a9eb58fa0044efe091109b4b209696764cdd0458e39c2166557da6900"}
Oct 31 13:26:29.688524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:29.687751    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj" event={"ID":"208fddf2-00ef-4f4a-944a-f3d93ba37911","Type":"ContainerStarted","Data":"68011f1f86b5e83e9ba011efb69bd13c889b66596544232fe3e37f10baa0af0d"}
Oct 31 13:26:29.713570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:29.713487    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h" podStartSLOduration=1.7134656129999999 podStartE2EDuration="1.713465613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:29.694096182 +0000 UTC m=+1387.092444845" watchObservedRunningTime="2026-10-31 13:26:29.713465613 +0000 UTC m=+1387.111814234"
Oct 31 13:26:29.731083 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:29.731025    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj" podStartSLOduration=1.731012329 podStartE2EDuration="1.731012329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:29.728240634 +0000 UTC m=+1387.126589259" watchObservedRunningTime="2026-10-31 13:26:29.731012329 +0000 UTC m=+1387.129360947"
Oct 31 13:26:29.774893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:29.769990    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:30.523889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:30.523819    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:30.528465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:30.528246    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:30.529599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:30.529566    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 31 13:26:30.529695 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:30.529634    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj" podUID="208fddf2-00ef-4f4a-944a-f3d93ba37911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 31 13:26:30.762896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:30.760814    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 13:26:30.762896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:30.760903    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h" podUID="a7c64ba1-f422-427b-80b8-1f464d32fafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 13:26:31.319303 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:31.319222    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:31.527917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:31.527852    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 31 13:26:31.528302 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:31.527953    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj" podUID="208fddf2-00ef-4f4a-944a-f3d93ba37911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 31 13:26:31.765894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:31.765222    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 13:26:31.765894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:31.765295    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h" podUID="a7c64ba1-f422-427b-80b8-1f464d32fafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 13:26:32.114792 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:32.114663    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:32.528165 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:32.528129    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 31 13:26:32.528601 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:32.528175    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj" podUID="208fddf2-00ef-4f4a-944a-f3d93ba37911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 31 13:26:32.767895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:32.767710    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 13:26:32.767895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:32.767764    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h" podUID="a7c64ba1-f422-427b-80b8-1f464d32fafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 13:26:32.930285 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:32.928631    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:33.528733 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:33.528702    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 31 13:26:33.529130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:33.528746    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj" podUID="208fddf2-00ef-4f4a-944a-f3d93ba37911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 31 13:26:33.763894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:33.760282    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 13:26:33.763894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:33.760344    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h" podUID="a7c64ba1-f422-427b-80b8-1f464d32fafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 13:26:33.822958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:33.817542    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:34.528690 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:34.528650    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 31 13:26:34.528934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:34.528709    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj" podUID="208fddf2-00ef-4f4a-944a-f3d93ba37911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 31 13:26:34.617894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:34.616410    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:34.769421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:34.769384    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 13:26:34.769717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:34.769677    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h" podUID="a7c64ba1-f422-427b-80b8-1f464d32fafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 13:26:35.500019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:35.499981    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:35.624704 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:35.624676    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:35.626068 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:35.626047    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:35.630176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:35.630154    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:26:35.871844 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:35.871768    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:35.872652 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:35.872618    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:35.873963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:35.873942    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 13:26:36.422604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:36.422559    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:37.216738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:37.216681    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:38.006969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:38.006922    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:38.902468 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:38.902393    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:39.814744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:39.814519    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:40.840890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:40.834668    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:40.906130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:40.906101    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"]
Oct 31 13:26:40.910100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:40.910085    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"
Oct 31 13:26:40.917710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:40.917687    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:26:40.917710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:40.917687    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:26:40.920978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:40.920950    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"]
Oct 31 13:26:40.988742 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:40.988716    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053fc722-bf18-4cc5-a5d2-884b8893a5a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98b7aeae-xgx25\" (UID: \"053fc722-bf18-4cc5-a5d2-884b8893a5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"
Oct 31 13:26:40.988919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:40.988748    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053fc722-bf18-4cc5-a5d2-884b8893a5a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98b7aeae-xgx25\" (UID: \"053fc722-bf18-4cc5-a5d2-884b8893a5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"
Oct 31 13:26:40.988919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:40.988771    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjszp\" (UniqueName: \"kubernetes.io/projected/053fc722-bf18-4cc5-a5d2-884b8893a5a8-kube-api-access-xjszp\") pod \"maas-api-revoke-sub-98b7aeae-xgx25\" (UID: \"053fc722-bf18-4cc5-a5d2-884b8893a5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"
Oct 31 13:26:41.089790 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:41.089761    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053fc722-bf18-4cc5-a5d2-884b8893a5a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98b7aeae-xgx25\" (UID: \"053fc722-bf18-4cc5-a5d2-884b8893a5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"
Oct 31 13:26:41.089790 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:41.089792    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053fc722-bf18-4cc5-a5d2-884b8893a5a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98b7aeae-xgx25\" (UID: \"053fc722-bf18-4cc5-a5d2-884b8893a5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"
Oct 31 13:26:41.090035 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:41.089812    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjszp\" (UniqueName: \"kubernetes.io/projected/053fc722-bf18-4cc5-a5d2-884b8893a5a8-kube-api-access-xjszp\") pod \"maas-api-revoke-sub-98b7aeae-xgx25\" (UID: \"053fc722-bf18-4cc5-a5d2-884b8893a5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"
Oct 31 13:26:41.090406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:41.090384    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053fc722-bf18-4cc5-a5d2-884b8893a5a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98b7aeae-xgx25\" (UID: \"053fc722-bf18-4cc5-a5d2-884b8893a5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"
Oct 31 13:26:41.092326 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:41.092283    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053fc722-bf18-4cc5-a5d2-884b8893a5a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98b7aeae-xgx25\" (UID: \"053fc722-bf18-4cc5-a5d2-884b8893a5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"
Oct 31 13:26:41.102557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:41.102532    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjszp\" (UniqueName: \"kubernetes.io/projected/053fc722-bf18-4cc5-a5d2-884b8893a5a8-kube-api-access-xjszp\") pod \"maas-api-revoke-sub-98b7aeae-xgx25\" (UID: \"053fc722-bf18-4cc5-a5d2-884b8893a5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"
Oct 31 13:26:41.225840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:41.225813    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"
Oct 31 13:26:41.505817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:41.505792    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"]
Oct 31 13:26:41.507578 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:26:41.507551    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053fc722_bf18_4cc5_a5d2_884b8893a5a8.slice/crio-01344051ccb26e773c6f6192538643b01fc0a8ac890724a5043696af6b1f1af1 WatchSource:0}: Error finding container 01344051ccb26e773c6f6192538643b01fc0a8ac890724a5043696af6b1f1af1: Status 404 returned error can't find the container with id 01344051ccb26e773c6f6192538643b01fc0a8ac890724a5043696af6b1f1af1
Oct 31 13:26:41.509595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:41.509575    4428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:26:41.616689 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:41.616651    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:41.771395 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:41.771312    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25" event={"ID":"053fc722-bf18-4cc5-a5d2-884b8893a5a8","Type":"ContainerStarted","Data":"01344051ccb26e773c6f6192538643b01fc0a8ac890724a5043696af6b1f1af1"}
Oct 31 13:26:42.583196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:42.583159    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:43.320765 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:43.320731    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:44.201721 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:44.201690    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:45.028068 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:45.028021    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:45.794983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:45.794950    4428 generic.go:356] "Generic (PLEG): container finished" podID="053fc722-bf18-4cc5-a5d2-884b8893a5a8" containerID="a7ff1e8370894abb741f297bf549badbd18312253c31234cbe21d00a1742297b" exitCode=0
Oct 31 13:26:45.795390 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:45.795042    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25" event={"ID":"053fc722-bf18-4cc5-a5d2-884b8893a5a8","Type":"ContainerDied","Data":"a7ff1e8370894abb741f297bf549badbd18312253c31234cbe21d00a1742297b"}
Oct 31 13:26:45.903620 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:45.903545    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:46.738890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:46.738294    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:46.955890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:46.955849    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"
Oct 31 13:26:46.981985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:46.981959    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/053fc722-bf18-4cc5-a5d2-884b8893a5a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053fc722-bf18-4cc5-a5d2-884b8893a5a8-maas-api-openshift-ca\") pod \"053fc722-bf18-4cc5-a5d2-884b8893a5a8\" (UID: \"053fc722-bf18-4cc5-a5d2-884b8893a5a8\") "
Oct 31 13:26:46.982113 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:46.982058    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/053fc722-bf18-4cc5-a5d2-884b8893a5a8-kube-api-access-xjszp\" (UniqueName: \"kubernetes.io/projected/053fc722-bf18-4cc5-a5d2-884b8893a5a8-kube-api-access-xjszp\") pod \"053fc722-bf18-4cc5-a5d2-884b8893a5a8\" (UID: \"053fc722-bf18-4cc5-a5d2-884b8893a5a8\") "
Oct 31 13:26:46.982178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:46.982166    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/053fc722-bf18-4cc5-a5d2-884b8893a5a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053fc722-bf18-4cc5-a5d2-884b8893a5a8-maas-api-xks-ca\") pod \"053fc722-bf18-4cc5-a5d2-884b8893a5a8\" (UID: \"053fc722-bf18-4cc5-a5d2-884b8893a5a8\") "
Oct 31 13:26:46.982307 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:46.982287    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/053fc722-bf18-4cc5-a5d2-884b8893a5a8-maas-api-openshift-ca" pod "053fc722-bf18-4cc5-a5d2-884b8893a5a8" (UID: "053fc722-bf18-4cc5-a5d2-884b8893a5a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:26:46.982579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:46.982561    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/053fc722-bf18-4cc5-a5d2-884b8893a5a8-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:26:46.984469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:46.984445    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053fc722-bf18-4cc5-a5d2-884b8893a5a8-kube-api-access-xjszp" pod "053fc722-bf18-4cc5-a5d2-884b8893a5a8" (UID: "053fc722-bf18-4cc5-a5d2-884b8893a5a8"). InnerVolumeSpecName "kube-api-access-xjszp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:26:46.984566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:46.984542    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/053fc722-bf18-4cc5-a5d2-884b8893a5a8-maas-api-xks-ca" pod "053fc722-bf18-4cc5-a5d2-884b8893a5a8" (UID: "053fc722-bf18-4cc5-a5d2-884b8893a5a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:26:47.083716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:47.083687    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/053fc722-bf18-4cc5-a5d2-884b8893a5a8-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:26:47.083716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:47.083717    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjszp\" (UniqueName: \"kubernetes.io/projected/053fc722-bf18-4cc5-a5d2-884b8893a5a8-kube-api-access-xjszp\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:26:47.608930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:47.608881    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:47.806760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:47.806728    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"
Oct 31 13:26:47.806957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:47.806721    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25" event={"ID":"053fc722-bf18-4cc5-a5d2-884b8893a5a8","Type":"ContainerDied","Data":"01344051ccb26e773c6f6192538643b01fc0a8ac890724a5043696af6b1f1af1"}
Oct 31 13:26:47.806957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:47.806861    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01344051ccb26e773c6f6192538643b01fc0a8ac890724a5043696af6b1f1af1"
Oct 31 13:26:48.417193 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:48.417152    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:48.740395 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:26:48.740362    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:26:49.206737 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:49.206660    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:50.101917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:50.101385    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:50.890357 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:50.890324    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:51.706498 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:51.706461    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:52.519030 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:52.518981    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:53.008883 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.008830    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"]
Oct 31 13:26:53.009578 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.009556    4428 cpu_manager.go:414] "containerMap: removing container" podUID="053fc722-bf18-4cc5-a5d2-884b8893a5a8" containerName="revoke-keys"
Oct 31 13:26:53.009683 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.009579    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="053fc722-bf18-4cc5-a5d2-884b8893a5a8" containerName="revoke-keys"
Oct 31 13:26:53.009774 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.009759    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="053fc722-bf18-4cc5-a5d2-884b8893a5a8" containerName="revoke-keys"
Oct 31 13:26:53.013290 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.013271    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"
Oct 31 13:26:53.021346 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.021324    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:26:53.021558 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.021390    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:26:53.024518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.024494    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"]
Oct 31 13:26:53.076234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.076206    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5f8r\" (UniqueName: \"kubernetes.io/projected/f1c06662-996a-4bf1-a02b-673daa51c549-kube-api-access-z5f8r\") pod \"maas-api-revoke-sub-02b2971e-wvdvz\" (UID: \"f1c06662-996a-4bf1-a02b-673daa51c549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"
Oct 31 13:26:53.076375 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.076254    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c06662-996a-4bf1-a02b-673daa51c549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b2971e-wvdvz\" (UID: \"f1c06662-996a-4bf1-a02b-673daa51c549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"
Oct 31 13:26:53.076416 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.076392    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c06662-996a-4bf1-a02b-673daa51c549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b2971e-wvdvz\" (UID: \"f1c06662-996a-4bf1-a02b-673daa51c549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"
Oct 31 13:26:53.177679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.177651    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c06662-996a-4bf1-a02b-673daa51c549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b2971e-wvdvz\" (UID: \"f1c06662-996a-4bf1-a02b-673daa51c549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"
Oct 31 13:26:53.177803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.177703    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5f8r\" (UniqueName: \"kubernetes.io/projected/f1c06662-996a-4bf1-a02b-673daa51c549-kube-api-access-z5f8r\") pod \"maas-api-revoke-sub-02b2971e-wvdvz\" (UID: \"f1c06662-996a-4bf1-a02b-673daa51c549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"
Oct 31 13:26:53.177803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.177734    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c06662-996a-4bf1-a02b-673daa51c549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b2971e-wvdvz\" (UID: \"f1c06662-996a-4bf1-a02b-673daa51c549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"
Oct 31 13:26:53.178357 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.178338    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c06662-996a-4bf1-a02b-673daa51c549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b2971e-wvdvz\" (UID: \"f1c06662-996a-4bf1-a02b-673daa51c549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"
Oct 31 13:26:53.180194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.180176    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c06662-996a-4bf1-a02b-673daa51c549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b2971e-wvdvz\" (UID: \"f1c06662-996a-4bf1-a02b-673daa51c549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"
Oct 31 13:26:53.191185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.191166    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5f8r\" (UniqueName: \"kubernetes.io/projected/f1c06662-996a-4bf1-a02b-673daa51c549-kube-api-access-z5f8r\") pod \"maas-api-revoke-sub-02b2971e-wvdvz\" (UID: \"f1c06662-996a-4bf1-a02b-673daa51c549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"
Oct 31 13:26:53.328938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.328126    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"
Oct 31 13:26:53.328938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.328781    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:53.616931 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.616909    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"]
Oct 31 13:26:53.618075 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:26:53.618051    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c06662_996a_4bf1_a02b_673daa51c549.slice/crio-210ea27ab5a6d08d3806951ebefedabbeb326fc2bb6ec66d4dee2cb08a75a297 WatchSource:0}: Error finding container 210ea27ab5a6d08d3806951ebefedabbeb326fc2bb6ec66d4dee2cb08a75a297: Status 404 returned error can't find the container with id 210ea27ab5a6d08d3806951ebefedabbeb326fc2bb6ec66d4dee2cb08a75a297
Oct 31 13:26:53.842466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:53.842435    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz" event={"ID":"f1c06662-996a-4bf1-a02b-673daa51c549","Type":"ContainerStarted","Data":"210ea27ab5a6d08d3806951ebefedabbeb326fc2bb6ec66d4dee2cb08a75a297"}
Oct 31 13:26:54.112889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:54.111705    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:54.850228 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:54.850145    4428 generic.go:356] "Generic (PLEG): container finished" podID="f1c06662-996a-4bf1-a02b-673daa51c549" containerID="c02c3693ec5c0433742a101a17836690b64fd4e9aae34a497803ad6572df2d44" exitCode=0
Oct 31 13:26:54.850228 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:54.850201    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz" event={"ID":"f1c06662-996a-4bf1-a02b-673daa51c549","Type":"ContainerDied","Data":"c02c3693ec5c0433742a101a17836690b64fd4e9aae34a497803ad6572df2d44"}
Oct 31 13:26:54.974497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:54.974458    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:55.719694 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:55.719608    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:56.009413 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:56.009389    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"
Oct 31 13:26:56.129082 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:56.129049    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c06662-996a-4bf1-a02b-673daa51c549-kube-api-access-z5f8r\" (UniqueName: \"kubernetes.io/projected/f1c06662-996a-4bf1-a02b-673daa51c549-kube-api-access-z5f8r\") pod \"f1c06662-996a-4bf1-a02b-673daa51c549\" (UID: \"f1c06662-996a-4bf1-a02b-673daa51c549\") "
Oct 31 13:26:56.129244 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:56.129099    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1c06662-996a-4bf1-a02b-673daa51c549-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c06662-996a-4bf1-a02b-673daa51c549-maas-api-openshift-ca\") pod \"f1c06662-996a-4bf1-a02b-673daa51c549\" (UID: \"f1c06662-996a-4bf1-a02b-673daa51c549\") "
Oct 31 13:26:56.129244 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:56.129137    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1c06662-996a-4bf1-a02b-673daa51c549-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c06662-996a-4bf1-a02b-673daa51c549-maas-api-xks-ca\") pod \"f1c06662-996a-4bf1-a02b-673daa51c549\" (UID: \"f1c06662-996a-4bf1-a02b-673daa51c549\") "
Oct 31 13:26:56.129532 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:56.129501    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c06662-996a-4bf1-a02b-673daa51c549-maas-api-openshift-ca" pod "f1c06662-996a-4bf1-a02b-673daa51c549" (UID: "f1c06662-996a-4bf1-a02b-673daa51c549"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:26:56.131416 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:56.131384    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c06662-996a-4bf1-a02b-673daa51c549-kube-api-access-z5f8r" pod "f1c06662-996a-4bf1-a02b-673daa51c549" (UID: "f1c06662-996a-4bf1-a02b-673daa51c549"). InnerVolumeSpecName "kube-api-access-z5f8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:26:56.131525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:56.131467    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1c06662-996a-4bf1-a02b-673daa51c549-maas-api-xks-ca" pod "f1c06662-996a-4bf1-a02b-673daa51c549" (UID: "f1c06662-996a-4bf1-a02b-673daa51c549"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:26:56.230985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:56.230950    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5f8r\" (UniqueName: \"kubernetes.io/projected/f1c06662-996a-4bf1-a02b-673daa51c549-kube-api-access-z5f8r\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:26:56.230985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:56.230979    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c06662-996a-4bf1-a02b-673daa51c549-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:26:56.230985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:56.230989    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c06662-996a-4bf1-a02b-673daa51c549-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:26:56.580812 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:56.580760    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:56.863132 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:56.863050    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz" event={"ID":"f1c06662-996a-4bf1-a02b-673daa51c549","Type":"ContainerDied","Data":"210ea27ab5a6d08d3806951ebefedabbeb326fc2bb6ec66d4dee2cb08a75a297"}
Oct 31 13:26:56.863132 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:56.863090    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"
Oct 31 13:26:56.863132 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:56.863096    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="210ea27ab5a6d08d3806951ebefedabbeb326fc2bb6ec66d4dee2cb08a75a297"
Oct 31 13:26:57.517891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.512037    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"]
Oct 31 13:26:57.517891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.512921    4428 cpu_manager.go:414] "containerMap: removing container" podUID="f1c06662-996a-4bf1-a02b-673daa51c549" containerName="revoke-keys"
Oct 31 13:26:57.517891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.512943    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c06662-996a-4bf1-a02b-673daa51c549" containerName="revoke-keys"
Oct 31 13:26:57.517891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.513132    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c06662-996a-4bf1-a02b-673daa51c549" containerName="revoke-keys"
Oct 31 13:26:57.521491 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.521452    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:57.525534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.525507    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"]
Oct 31 13:26:57.525920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.525902    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"
Oct 31 13:26:57.535903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.532490    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:26:57.535903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.533158    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:26:57.646688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.646659    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73586db-b1bb-4629-9357-aacc597c436f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f692a8f5-d8dqm\" (UID: \"a73586db-b1bb-4629-9357-aacc597c436f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"
Oct 31 13:26:57.646923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.646903    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5brc\" (UniqueName: \"kubernetes.io/projected/a73586db-b1bb-4629-9357-aacc597c436f-kube-api-access-h5brc\") pod \"maas-api-revoke-sub-f692a8f5-d8dqm\" (UID: \"a73586db-b1bb-4629-9357-aacc597c436f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"
Oct 31 13:26:57.647011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.646941    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73586db-b1bb-4629-9357-aacc597c436f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f692a8f5-d8dqm\" (UID: \"a73586db-b1bb-4629-9357-aacc597c436f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"
Oct 31 13:26:57.747503 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.747469    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5brc\" (UniqueName: \"kubernetes.io/projected/a73586db-b1bb-4629-9357-aacc597c436f-kube-api-access-h5brc\") pod \"maas-api-revoke-sub-f692a8f5-d8dqm\" (UID: \"a73586db-b1bb-4629-9357-aacc597c436f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"
Oct 31 13:26:57.747614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.747519    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73586db-b1bb-4629-9357-aacc597c436f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f692a8f5-d8dqm\" (UID: \"a73586db-b1bb-4629-9357-aacc597c436f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"
Oct 31 13:26:57.747614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.747562    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73586db-b1bb-4629-9357-aacc597c436f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f692a8f5-d8dqm\" (UID: \"a73586db-b1bb-4629-9357-aacc597c436f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"
Oct 31 13:26:57.748250 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.748197    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73586db-b1bb-4629-9357-aacc597c436f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f692a8f5-d8dqm\" (UID: \"a73586db-b1bb-4629-9357-aacc597c436f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"
Oct 31 13:26:57.749982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.749962    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73586db-b1bb-4629-9357-aacc597c436f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f692a8f5-d8dqm\" (UID: \"a73586db-b1bb-4629-9357-aacc597c436f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"
Oct 31 13:26:57.790892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.790793    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5brc\" (UniqueName: \"kubernetes.io/projected/a73586db-b1bb-4629-9357-aacc597c436f-kube-api-access-h5brc\") pod \"maas-api-revoke-sub-f692a8f5-d8dqm\" (UID: \"a73586db-b1bb-4629-9357-aacc597c436f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"
Oct 31 13:26:57.847093 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:57.847057    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"
Oct 31 13:26:58.149995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:58.149964    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"]
Oct 31 13:26:58.150993 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:26:58.150962    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73586db_b1bb_4629_9357_aacc597c436f.slice/crio-cb01155ba7d769a26f057ca8b20f2eb8a4fbbb1b94aac05410b2aea2f3282b0d WatchSource:0}: Error finding container cb01155ba7d769a26f057ca8b20f2eb8a4fbbb1b94aac05410b2aea2f3282b0d: Status 404 returned error can't find the container with id cb01155ba7d769a26f057ca8b20f2eb8a4fbbb1b94aac05410b2aea2f3282b0d
Oct 31 13:26:58.570892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:58.568421    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:58.879998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:58.879964    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm" event={"ID":"a73586db-b1bb-4629-9357-aacc597c436f","Type":"ContainerStarted","Data":"e71d4a4910aaa853ba618375921cdba12ded1713b76de349a16af9d6d0bc36fb"}
Oct 31 13:26:58.879998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:58.880016    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm" event={"ID":"a73586db-b1bb-4629-9357-aacc597c436f","Type":"ContainerStarted","Data":"cb01155ba7d769a26f057ca8b20f2eb8a4fbbb1b94aac05410b2aea2f3282b0d"}
Oct 31 13:26:58.902057 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:58.902006    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm" podStartSLOduration=1.9019912749999999 podStartE2EDuration="1.901991275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:58.899293027 +0000 UTC m=+1416.297641651" watchObservedRunningTime="2026-10-31 13:26:58.901991275 +0000 UTC m=+1416.300339930"
Oct 31 13:26:59.375372 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:59.375341    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"]
Oct 31 13:26:59.385218 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:59.385170    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"
Oct 31 13:26:59.416910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:59.416501    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:26:59.425398 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:59.425321    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"]
Oct 31 13:26:59.474317 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:59.474282    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c69f8f7-f106-4c54-ad97-616218cb4ab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78cc6525-gtvcn\" (UID: \"0c69f8f7-f106-4c54-ad97-616218cb4ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"
Oct 31 13:26:59.474472 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:59.474394    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c69f8f7-f106-4c54-ad97-616218cb4ab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78cc6525-gtvcn\" (UID: \"0c69f8f7-f106-4c54-ad97-616218cb4ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"
Oct 31 13:26:59.474472 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:59.474455    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqqmq\" (UniqueName: \"kubernetes.io/projected/0c69f8f7-f106-4c54-ad97-616218cb4ab8-kube-api-access-pqqmq\") pod \"maas-api-revoke-sub-78cc6525-gtvcn\" (UID: \"0c69f8f7-f106-4c54-ad97-616218cb4ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"
Oct 31 13:26:59.575675 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:59.575647    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c69f8f7-f106-4c54-ad97-616218cb4ab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78cc6525-gtvcn\" (UID: \"0c69f8f7-f106-4c54-ad97-616218cb4ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"
Oct 31 13:26:59.575821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:59.575692    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqqmq\" (UniqueName: \"kubernetes.io/projected/0c69f8f7-f106-4c54-ad97-616218cb4ab8-kube-api-access-pqqmq\") pod \"maas-api-revoke-sub-78cc6525-gtvcn\" (UID: \"0c69f8f7-f106-4c54-ad97-616218cb4ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"
Oct 31 13:26:59.575821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:59.575778    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c69f8f7-f106-4c54-ad97-616218cb4ab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78cc6525-gtvcn\" (UID: \"0c69f8f7-f106-4c54-ad97-616218cb4ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"
Oct 31 13:26:59.585082 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:59.576908    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c69f8f7-f106-4c54-ad97-616218cb4ab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78cc6525-gtvcn\" (UID: \"0c69f8f7-f106-4c54-ad97-616218cb4ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"
Oct 31 13:26:59.586074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:59.586051    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c69f8f7-f106-4c54-ad97-616218cb4ab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78cc6525-gtvcn\" (UID: \"0c69f8f7-f106-4c54-ad97-616218cb4ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"
Oct 31 13:26:59.589375 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:59.589350    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqqmq\" (UniqueName: \"kubernetes.io/projected/0c69f8f7-f106-4c54-ad97-616218cb4ab8-kube-api-access-pqqmq\") pod \"maas-api-revoke-sub-78cc6525-gtvcn\" (UID: \"0c69f8f7-f106-4c54-ad97-616218cb4ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"
Oct 31 13:26:59.722742 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:59.722715    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"
Oct 31 13:26:59.886842 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:59.886807    4428 generic.go:356] "Generic (PLEG): container finished" podID="a73586db-b1bb-4629-9357-aacc597c436f" containerID="e71d4a4910aaa853ba618375921cdba12ded1713b76de349a16af9d6d0bc36fb" exitCode=0
Oct 31 13:26:59.887005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:26:59.886900    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm" event={"ID":"a73586db-b1bb-4629-9357-aacc597c436f","Type":"ContainerDied","Data":"e71d4a4910aaa853ba618375921cdba12ded1713b76de349a16af9d6d0bc36fb"}
Oct 31 13:27:00.001890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:00.001858    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"]
Oct 31 13:27:00.003434 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:27:00.003405    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c69f8f7_f106_4c54_ad97_616218cb4ab8.slice/crio-39d9cdea0cd0643b407f6a9f01163cd42d519c6a251be397d0c97509a198274a WatchSource:0}: Error finding container 39d9cdea0cd0643b407f6a9f01163cd42d519c6a251be397d0c97509a198274a: Status 404 returned error can't find the container with id 39d9cdea0cd0643b407f6a9f01163cd42d519c6a251be397d0c97509a198274a
Oct 31 13:27:00.126378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:00.126327    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:00.894084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:00.894055    4428 generic.go:356] "Generic (PLEG): container finished" podID="0c69f8f7-f106-4c54-ad97-616218cb4ab8" containerID="e74cfb2ad407bc5147012b7155e9e655fe04864c2e1c84229e3a8644502bbea8" exitCode=0
Oct 31 13:27:00.894490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:00.894141    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn" event={"ID":"0c69f8f7-f106-4c54-ad97-616218cb4ab8","Type":"ContainerDied","Data":"e74cfb2ad407bc5147012b7155e9e655fe04864c2e1c84229e3a8644502bbea8"}
Oct 31 13:27:00.894490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:00.894196    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn" event={"ID":"0c69f8f7-f106-4c54-ad97-616218cb4ab8","Type":"ContainerStarted","Data":"39d9cdea0cd0643b407f6a9f01163cd42d519c6a251be397d0c97509a198274a"}
Oct 31 13:27:00.942916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:00.937267    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:01.068689 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:01.068663    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"
Oct 31 13:27:01.195553 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:01.195524    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a73586db-b1bb-4629-9357-aacc597c436f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73586db-b1bb-4629-9357-aacc597c436f-maas-api-xks-ca\") pod \"a73586db-b1bb-4629-9357-aacc597c436f\" (UID: \"a73586db-b1bb-4629-9357-aacc597c436f\") "
Oct 31 13:27:01.195746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:01.195603    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a73586db-b1bb-4629-9357-aacc597c436f-kube-api-access-h5brc\" (UniqueName: \"kubernetes.io/projected/a73586db-b1bb-4629-9357-aacc597c436f-kube-api-access-h5brc\") pod \"a73586db-b1bb-4629-9357-aacc597c436f\" (UID: \"a73586db-b1bb-4629-9357-aacc597c436f\") "
Oct 31 13:27:01.195746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:01.195657    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a73586db-b1bb-4629-9357-aacc597c436f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73586db-b1bb-4629-9357-aacc597c436f-maas-api-openshift-ca\") pod \"a73586db-b1bb-4629-9357-aacc597c436f\" (UID: \"a73586db-b1bb-4629-9357-aacc597c436f\") "
Oct 31 13:27:01.196083 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:01.196058    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73586db-b1bb-4629-9357-aacc597c436f-maas-api-openshift-ca" pod "a73586db-b1bb-4629-9357-aacc597c436f" (UID: "a73586db-b1bb-4629-9357-aacc597c436f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:27:01.196224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:01.196210    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73586db-b1bb-4629-9357-aacc597c436f-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:01.197839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:01.197809    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a73586db-b1bb-4629-9357-aacc597c436f-maas-api-xks-ca" pod "a73586db-b1bb-4629-9357-aacc597c436f" (UID: "a73586db-b1bb-4629-9357-aacc597c436f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:27:01.197954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:01.197933    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73586db-b1bb-4629-9357-aacc597c436f-kube-api-access-h5brc" pod "a73586db-b1bb-4629-9357-aacc597c436f" (UID: "a73586db-b1bb-4629-9357-aacc597c436f"). InnerVolumeSpecName "kube-api-access-h5brc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:27:01.297137 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:01.297072    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73586db-b1bb-4629-9357-aacc597c436f-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:01.297137 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:01.297096    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5brc\" (UniqueName: \"kubernetes.io/projected/a73586db-b1bb-4629-9357-aacc597c436f-kube-api-access-h5brc\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:01.809663 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:01.809619    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:01.901427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:01.901401    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"
Oct 31 13:27:01.901810 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:01.901460    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm" event={"ID":"a73586db-b1bb-4629-9357-aacc597c436f","Type":"ContainerDied","Data":"cb01155ba7d769a26f057ca8b20f2eb8a4fbbb1b94aac05410b2aea2f3282b0d"}
Oct 31 13:27:01.901810 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:01.901498    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb01155ba7d769a26f057ca8b20f2eb8a4fbbb1b94aac05410b2aea2f3282b0d"
Oct 31 13:27:02.054108 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:02.054082    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"
Oct 31 13:27:02.210171 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:02.210149    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c69f8f7-f106-4c54-ad97-616218cb4ab8-kube-api-access-pqqmq\" (UniqueName: \"kubernetes.io/projected/0c69f8f7-f106-4c54-ad97-616218cb4ab8-kube-api-access-pqqmq\") pod \"0c69f8f7-f106-4c54-ad97-616218cb4ab8\" (UID: \"0c69f8f7-f106-4c54-ad97-616218cb4ab8\") "
Oct 31 13:27:02.210308 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:02.210242    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c69f8f7-f106-4c54-ad97-616218cb4ab8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c69f8f7-f106-4c54-ad97-616218cb4ab8-maas-api-openshift-ca\") pod \"0c69f8f7-f106-4c54-ad97-616218cb4ab8\" (UID: \"0c69f8f7-f106-4c54-ad97-616218cb4ab8\") "
Oct 31 13:27:02.210308 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:02.210297    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c69f8f7-f106-4c54-ad97-616218cb4ab8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c69f8f7-f106-4c54-ad97-616218cb4ab8-maas-api-xks-ca\") pod \"0c69f8f7-f106-4c54-ad97-616218cb4ab8\" (UID: \"0c69f8f7-f106-4c54-ad97-616218cb4ab8\") "
Oct 31 13:27:02.210617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:02.210592    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c69f8f7-f106-4c54-ad97-616218cb4ab8-maas-api-openshift-ca" pod "0c69f8f7-f106-4c54-ad97-616218cb4ab8" (UID: "0c69f8f7-f106-4c54-ad97-616218cb4ab8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:27:02.210790 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:02.210776    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c69f8f7-f106-4c54-ad97-616218cb4ab8-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:02.212504 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:02.212469    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c69f8f7-f106-4c54-ad97-616218cb4ab8-maas-api-xks-ca" pod "0c69f8f7-f106-4c54-ad97-616218cb4ab8" (UID: "0c69f8f7-f106-4c54-ad97-616218cb4ab8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:27:02.212607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:02.212582    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c69f8f7-f106-4c54-ad97-616218cb4ab8-kube-api-access-pqqmq" pod "0c69f8f7-f106-4c54-ad97-616218cb4ab8" (UID: "0c69f8f7-f106-4c54-ad97-616218cb4ab8"). InnerVolumeSpecName "kube-api-access-pqqmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:27:02.312125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:02.312092    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c69f8f7-f106-4c54-ad97-616218cb4ab8-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:02.312125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:02.312118    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqqmq\" (UniqueName: \"kubernetes.io/projected/0c69f8f7-f106-4c54-ad97-616218cb4ab8-kube-api-access-pqqmq\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:02.603857 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:02.601897    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:02.908627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:02.908551    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"
Oct 31 13:27:02.908627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:02.908559    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn" event={"ID":"0c69f8f7-f106-4c54-ad97-616218cb4ab8","Type":"ContainerDied","Data":"39d9cdea0cd0643b407f6a9f01163cd42d519c6a251be397d0c97509a198274a"}
Oct 31 13:27:02.908627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:02.908624    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39d9cdea0cd0643b407f6a9f01163cd42d519c6a251be397d0c97509a198274a"
Oct 31 13:27:03.386153 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:03.386101    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:04.308554 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:04.308512    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:05.123893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:05.121543    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:06.012024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:06.005271    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:06.858892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:06.851123    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:06.974826 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:06.974793    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"]
Oct 31 13:27:06.975301 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:06.975286    4428 cpu_manager.go:414] "containerMap: removing container" podUID="a73586db-b1bb-4629-9357-aacc597c436f" containerName="revoke-keys"
Oct 31 13:27:06.975301 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:06.975300    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a73586db-b1bb-4629-9357-aacc597c436f" containerName="revoke-keys"
Oct 31 13:27:06.975397 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:06.975311    4428 cpu_manager.go:414] "containerMap: removing container" podUID="0c69f8f7-f106-4c54-ad97-616218cb4ab8" containerName="revoke-keys"
Oct 31 13:27:06.975397 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:06.975316    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c69f8f7-f106-4c54-ad97-616218cb4ab8" containerName="revoke-keys"
Oct 31 13:27:06.975471 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:06.975407    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a73586db-b1bb-4629-9357-aacc597c436f" containerName="revoke-keys"
Oct 31 13:27:06.975471 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:06.975416    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c69f8f7-f106-4c54-ad97-616218cb4ab8" containerName="revoke-keys"
Oct 31 13:27:06.980994 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:06.980965    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"
Oct 31 13:27:06.986108 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:06.986083    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:27:06.986250 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:06.986212    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:27:06.990223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:06.989823    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"]
Oct 31 13:27:07.169026 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:07.168942    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg2bd\" (UniqueName: \"kubernetes.io/projected/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-kube-api-access-xg2bd\") pod \"maas-api-revoke-sub-69f67ed7-z55vx\" (UID: \"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"
Oct 31 13:27:07.169026 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:07.169005    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69f67ed7-z55vx\" (UID: \"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"
Oct 31 13:27:07.169506 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:07.169266    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69f67ed7-z55vx\" (UID: \"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"
Oct 31 13:27:07.270751 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:07.270727    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69f67ed7-z55vx\" (UID: \"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"
Oct 31 13:27:07.270916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:07.270820    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69f67ed7-z55vx\" (UID: \"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"
Oct 31 13:27:07.270916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:07.270848    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg2bd\" (UniqueName: \"kubernetes.io/projected/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-kube-api-access-xg2bd\") pod \"maas-api-revoke-sub-69f67ed7-z55vx\" (UID: \"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"
Oct 31 13:27:07.271507 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:07.271482    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69f67ed7-z55vx\" (UID: \"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"
Oct 31 13:27:07.273306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:07.273282    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69f67ed7-z55vx\" (UID: \"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"
Oct 31 13:27:07.284978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:07.284952    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg2bd\" (UniqueName: \"kubernetes.io/projected/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-kube-api-access-xg2bd\") pod \"maas-api-revoke-sub-69f67ed7-z55vx\" (UID: \"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"
Oct 31 13:27:07.295767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:07.295745    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"
Oct 31 13:27:07.583174 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:07.583142    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"]
Oct 31 13:27:07.583358 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:27:07.583332    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb1ec4c_74bd_4137_a4c5_a3ce25638f21.slice/crio-51c12b6c96621ae76579d1388a65d9b32229e352d7c5dd74b68b67cfe0b5588a WatchSource:0}: Error finding container 51c12b6c96621ae76579d1388a65d9b32229e352d7c5dd74b68b67cfe0b5588a: Status 404 returned error can't find the container with id 51c12b6c96621ae76579d1388a65d9b32229e352d7c5dd74b68b67cfe0b5588a
Oct 31 13:27:07.627344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:07.625155    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:07.939612 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:07.939576    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx" event={"ID":"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21","Type":"ContainerStarted","Data":"51c12b6c96621ae76579d1388a65d9b32229e352d7c5dd74b68b67cfe0b5588a"}
Oct 31 13:27:08.548592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:08.534457    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:08.889360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:08.889273    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:08.972890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:08.972754    4428 generic.go:356] "Generic (PLEG): container finished" podID="bdb1ec4c-74bd-4137-a4c5-a3ce25638f21" containerID="357b4a2bb43dc82170a490919e3acefb5d955b5e1c744bdf2324e5883e3125c5" exitCode=0
Oct 31 13:27:08.973077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:08.972896    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx" event={"ID":"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21","Type":"ContainerDied","Data":"357b4a2bb43dc82170a490919e3acefb5d955b5e1c744bdf2324e5883e3125c5"}
Oct 31 13:27:09.578034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:09.578004    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:10.164705 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:10.164682    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"
Oct 31 13:27:10.207893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:10.207850    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-maas-api-openshift-ca\") pod \"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21\" (UID: \"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21\") "
Oct 31 13:27:10.208072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:10.207916    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-maas-api-xks-ca\") pod \"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21\" (UID: \"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21\") "
Oct 31 13:27:10.208146 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:10.208094    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-kube-api-access-xg2bd\" (UniqueName: \"kubernetes.io/projected/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-kube-api-access-xg2bd\") pod \"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21\" (UID: \"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21\") "
Oct 31 13:27:10.208228 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:10.208206    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-maas-api-openshift-ca" pod "bdb1ec4c-74bd-4137-a4c5-a3ce25638f21" (UID: "bdb1ec4c-74bd-4137-a4c5-a3ce25638f21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:27:10.208460 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:10.208427    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:10.210288 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:10.210262    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-maas-api-xks-ca" pod "bdb1ec4c-74bd-4137-a4c5-a3ce25638f21" (UID: "bdb1ec4c-74bd-4137-a4c5-a3ce25638f21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:27:10.210396 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:10.210288    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-kube-api-access-xg2bd" pod "bdb1ec4c-74bd-4137-a4c5-a3ce25638f21" (UID: "bdb1ec4c-74bd-4137-a4c5-a3ce25638f21"). InnerVolumeSpecName "kube-api-access-xg2bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:27:10.309445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:10.309366    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg2bd\" (UniqueName: \"kubernetes.io/projected/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-kube-api-access-xg2bd\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:10.309445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:10.309392    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:10.531892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:10.528528    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:10.988893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:10.988557    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"
Oct 31 13:27:10.988893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:10.988556    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx" event={"ID":"bdb1ec4c-74bd-4137-a4c5-a3ce25638f21","Type":"ContainerDied","Data":"51c12b6c96621ae76579d1388a65d9b32229e352d7c5dd74b68b67cfe0b5588a"}
Oct 31 13:27:10.988893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:10.988687    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51c12b6c96621ae76579d1388a65d9b32229e352d7c5dd74b68b67cfe0b5588a"
Oct 31 13:27:11.964465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:11.964432    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:12.281449 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.281369    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"]
Oct 31 13:27:12.282732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.282709    4428 cpu_manager.go:414] "containerMap: removing container" podUID="bdb1ec4c-74bd-4137-a4c5-a3ce25638f21" containerName="revoke-keys"
Oct 31 13:27:12.282889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.282857    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdb1ec4c-74bd-4137-a4c5-a3ce25638f21" containerName="revoke-keys"
Oct 31 13:27:12.283121 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.283110    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdb1ec4c-74bd-4137-a4c5-a3ce25638f21" containerName="revoke-keys"
Oct 31 13:27:12.313060 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.286936    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"]
Oct 31 13:27:12.313060 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.287044    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"
Oct 31 13:27:12.316853 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.314903    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:27:12.316853 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.316011    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:27:12.330901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.327014    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:12.439389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.439348    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fbpg\" (UniqueName: \"kubernetes.io/projected/3f336286-4447-42e0-98be-cb39b7f0a4ee-kube-api-access-9fbpg\") pod \"maas-api-revoke-sub-807497f4-nh5x5\" (UID: \"3f336286-4447-42e0-98be-cb39b7f0a4ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"
Oct 31 13:27:12.439717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.439696    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f336286-4447-42e0-98be-cb39b7f0a4ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-807497f4-nh5x5\" (UID: \"3f336286-4447-42e0-98be-cb39b7f0a4ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"
Oct 31 13:27:12.439918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.439898    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f336286-4447-42e0-98be-cb39b7f0a4ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-807497f4-nh5x5\" (UID: \"3f336286-4447-42e0-98be-cb39b7f0a4ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"
Oct 31 13:27:12.543735 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.543613    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fbpg\" (UniqueName: \"kubernetes.io/projected/3f336286-4447-42e0-98be-cb39b7f0a4ee-kube-api-access-9fbpg\") pod \"maas-api-revoke-sub-807497f4-nh5x5\" (UID: \"3f336286-4447-42e0-98be-cb39b7f0a4ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"
Oct 31 13:27:12.544089 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.544067    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f336286-4447-42e0-98be-cb39b7f0a4ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-807497f4-nh5x5\" (UID: \"3f336286-4447-42e0-98be-cb39b7f0a4ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"
Oct 31 13:27:12.544288 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.544262    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f336286-4447-42e0-98be-cb39b7f0a4ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-807497f4-nh5x5\" (UID: \"3f336286-4447-42e0-98be-cb39b7f0a4ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"
Oct 31 13:27:12.546134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.546096    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f336286-4447-42e0-98be-cb39b7f0a4ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-807497f4-nh5x5\" (UID: \"3f336286-4447-42e0-98be-cb39b7f0a4ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"
Oct 31 13:27:12.555043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.555007    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f336286-4447-42e0-98be-cb39b7f0a4ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-807497f4-nh5x5\" (UID: \"3f336286-4447-42e0-98be-cb39b7f0a4ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"
Oct 31 13:27:12.563400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.563344    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fbpg\" (UniqueName: \"kubernetes.io/projected/3f336286-4447-42e0-98be-cb39b7f0a4ee-kube-api-access-9fbpg\") pod \"maas-api-revoke-sub-807497f4-nh5x5\" (UID: \"3f336286-4447-42e0-98be-cb39b7f0a4ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"
Oct 31 13:27:12.770023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:12.769978    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"
Oct 31 13:27:13.175563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.175471    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:13.430006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.429955    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"]
Oct 31 13:27:13.430992 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:27:13.430964    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f336286_4447_42e0_98be_cb39b7f0a4ee.slice/crio-71bddc30a9d5ee830d76dcc1edcc03b3b83428ec282cbdd59c15540b35436f83 WatchSource:0}: Error finding container 71bddc30a9d5ee830d76dcc1edcc03b3b83428ec282cbdd59c15540b35436f83: Status 404 returned error can't find the container with id 71bddc30a9d5ee830d76dcc1edcc03b3b83428ec282cbdd59c15540b35436f83
Oct 31 13:27:13.529899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.525924    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"]
Oct 31 13:27:13.537656 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.537640    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.639502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.639456    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.639694 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.639523    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.639694 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.639570    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.639694 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.639620    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb96175b-4567-4467-a2fa-1aacfeffa985-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.639916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.639746    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.639977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.639932    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bplr\" (UniqueName: \"kubernetes.io/projected/fb96175b-4567-4467-a2fa-1aacfeffa985-kube-api-access-9bplr\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.640564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.640062    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.640564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.640098    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.640564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.640136    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.739631 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.739602    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"]
Oct 31 13:27:13.741486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.741461    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.741682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.741667    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bplr\" (UniqueName: \"kubernetes.io/projected/fb96175b-4567-4467-a2fa-1aacfeffa985-kube-api-access-9bplr\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.741828 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.741817    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.741937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.741925    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.742028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.742018    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.742352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.742337    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.743249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.743225    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.743334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.743269    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.743334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.743304    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb96175b-4567-4467-a2fa-1aacfeffa985-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.743434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.742335    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.743484 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.743467    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.743531 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.743509    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.743658 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.743638    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.744207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.744187    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb96175b-4567-4467-a2fa-1aacfeffa985-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.768012 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.745052    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.783432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.783399    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.826897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.791617    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bplr\" (UniqueName: \"kubernetes.io/projected/fb96175b-4567-4467-a2fa-1aacfeffa985-kube-api-access-9bplr\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.827088 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.811456    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:13.940332 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:13.940304    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:14.060893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:14.046839    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5" event={"ID":"3f336286-4447-42e0-98be-cb39b7f0a4ee","Type":"ContainerStarted","Data":"e58f3fcc4a84107059728062ef52982dfcf8e1416083104f9b3c3ef1d7f18517"}
Oct 31 13:27:14.060893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:14.046934    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5" event={"ID":"3f336286-4447-42e0-98be-cb39b7f0a4ee","Type":"ContainerStarted","Data":"71bddc30a9d5ee830d76dcc1edcc03b3b83428ec282cbdd59c15540b35436f83"}
Oct 31 13:27:14.094920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:14.073151    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5" podStartSLOduration=2.073132205 podStartE2EDuration="2.073132205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:14.072090355 +0000 UTC m=+1431.470438999" watchObservedRunningTime="2026-10-31 13:27:14.073132205 +0000 UTC m=+1431.471480831"
Oct 31 13:27:14.423437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:14.423350    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:14.428889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:14.426055    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"]
Oct 31 13:27:14.475896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:14.472251    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:27:14.475896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:14.472346    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:27:14.475896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:14.472390    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:27:15.055367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:15.055321    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb" event={"ID":"fb96175b-4567-4467-a2fa-1aacfeffa985","Type":"ContainerStarted","Data":"be4d146af5e6902055d82ab6c3d5e13ffc97820438b61536634b0dbc8f7bde20"}
Oct 31 13:27:15.055572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:15.055393    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb" event={"ID":"fb96175b-4567-4467-a2fa-1aacfeffa985","Type":"ContainerStarted","Data":"f6e14f2219576301478294e84ccd83311ad1afb20e367ff45b7e08d32a34db64"}
Oct 31 13:27:15.057240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:15.057211    4428 generic.go:356] "Generic (PLEG): container finished" podID="3f336286-4447-42e0-98be-cb39b7f0a4ee" containerID="e58f3fcc4a84107059728062ef52982dfcf8e1416083104f9b3c3ef1d7f18517" exitCode=0
Oct 31 13:27:15.057355 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:15.057252    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5" event={"ID":"3f336286-4447-42e0-98be-cb39b7f0a4ee","Type":"ContainerDied","Data":"e58f3fcc4a84107059728062ef52982dfcf8e1416083104f9b3c3ef1d7f18517"}
Oct 31 13:27:15.095665 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:15.095599    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb" podStartSLOduration=2.095576142 podStartE2EDuration="2.095576142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:15.089393589 +0000 UTC m=+1432.487742213" watchObservedRunningTime="2026-10-31 13:27:15.095576142 +0000 UTC m=+1432.493924768"
Oct 31 13:27:15.218937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:15.218559    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:15.941128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:15.941098    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:15.942850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:15.942826    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 13:27:15.943000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:15.942889    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb" podUID="fb96175b-4567-4467-a2fa-1aacfeffa985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 13:27:16.216945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:16.215771    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:16.315599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:16.315573    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"
Oct 31 13:27:16.442548 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:16.442513    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f336286-4447-42e0-98be-cb39b7f0a4ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f336286-4447-42e0-98be-cb39b7f0a4ee-maas-api-xks-ca\") pod \"3f336286-4447-42e0-98be-cb39b7f0a4ee\" (UID: \"3f336286-4447-42e0-98be-cb39b7f0a4ee\") "
Oct 31 13:27:16.442717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:16.442560    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f336286-4447-42e0-98be-cb39b7f0a4ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f336286-4447-42e0-98be-cb39b7f0a4ee-maas-api-openshift-ca\") pod \"3f336286-4447-42e0-98be-cb39b7f0a4ee\" (UID: \"3f336286-4447-42e0-98be-cb39b7f0a4ee\") "
Oct 31 13:27:16.442717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:16.442684    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f336286-4447-42e0-98be-cb39b7f0a4ee-kube-api-access-9fbpg\" (UniqueName: \"kubernetes.io/projected/3f336286-4447-42e0-98be-cb39b7f0a4ee-kube-api-access-9fbpg\") pod \"3f336286-4447-42e0-98be-cb39b7f0a4ee\" (UID: \"3f336286-4447-42e0-98be-cb39b7f0a4ee\") "
Oct 31 13:27:16.443048 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:16.443011    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f336286-4447-42e0-98be-cb39b7f0a4ee-maas-api-openshift-ca" pod "3f336286-4447-42e0-98be-cb39b7f0a4ee" (UID: "3f336286-4447-42e0-98be-cb39b7f0a4ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:27:16.470909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:16.470820    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f336286-4447-42e0-98be-cb39b7f0a4ee-kube-api-access-9fbpg" pod "3f336286-4447-42e0-98be-cb39b7f0a4ee" (UID: "3f336286-4447-42e0-98be-cb39b7f0a4ee"). InnerVolumeSpecName "kube-api-access-9fbpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:27:16.477019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:16.476982    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f336286-4447-42e0-98be-cb39b7f0a4ee-maas-api-xks-ca" pod "3f336286-4447-42e0-98be-cb39b7f0a4ee" (UID: "3f336286-4447-42e0-98be-cb39b7f0a4ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:27:16.544299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:16.544275    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fbpg\" (UniqueName: \"kubernetes.io/projected/3f336286-4447-42e0-98be-cb39b7f0a4ee-kube-api-access-9fbpg\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:16.544299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:16.544300    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f336286-4447-42e0-98be-cb39b7f0a4ee-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:16.544463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:16.544311    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f336286-4447-42e0-98be-cb39b7f0a4ee-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:16.942056 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:16.942023    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 13:27:16.942443 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:16.942078    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb" podUID="fb96175b-4567-4467-a2fa-1aacfeffa985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 13:27:17.081944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:17.077638    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:17.117528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:17.117491    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"
Oct 31 13:27:17.117691 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:17.117521    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5" event={"ID":"3f336286-4447-42e0-98be-cb39b7f0a4ee","Type":"ContainerDied","Data":"71bddc30a9d5ee830d76dcc1edcc03b3b83428ec282cbdd59c15540b35436f83"}
Oct 31 13:27:17.117691 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:17.117598    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71bddc30a9d5ee830d76dcc1edcc03b3b83428ec282cbdd59c15540b35436f83"
Oct 31 13:27:17.941522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:17.941491    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 13:27:17.941710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:17.941549    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb" podUID="fb96175b-4567-4467-a2fa-1aacfeffa985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 13:27:18.106930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:18.105802    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:18.740441 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:27:18.740406    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:27:18.942089 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:18.942059    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 13:27:18.942231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:18.942111    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb" podUID="fb96175b-4567-4467-a2fa-1aacfeffa985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 13:27:19.071150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:19.071054    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:19.941171 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:19.941135    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 13:27:19.941614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:19.941184    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb" podUID="fb96175b-4567-4467-a2fa-1aacfeffa985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 13:27:20.081607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:20.081574    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:20.249933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:20.249212    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:20.941536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:20.941504    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 13:27:20.941944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:20.941554    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb" podUID="fb96175b-4567-4467-a2fa-1aacfeffa985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 13:27:21.034818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:21.034788    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:21.453716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:21.453684    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:21.646135 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:21.646101    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:21.848194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:21.848163    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:21.945958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:21.945933    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:21.946329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:21.946221    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:21.946906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:21.946888    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:27:22.685361 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:22.685317    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:23.518346 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:23.518253    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:24.323037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:24.323005    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:25.114365 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:25.114264    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:26.004585 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:26.004521    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:27.229889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:27.225331    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:28.072558 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:28.065391    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:28.267991 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:28.267948    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:27:28.438126 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:28.436315    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:27:28.447919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:28.447050    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:27:28.475571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:28.475513    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:27:28.865412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:28.865371    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:32.082699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:32.082664    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:32.256767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:32.256736    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:32.879928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:32.879891    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:33.972897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:33.972568    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:34.960548 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:34.960514    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:35.917536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:35.917424    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:36.715829 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:36.715699    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:36.828105 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:36.828068    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"]
Oct 31 13:27:36.828941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:36.828919    4428 cpu_manager.go:414] "containerMap: removing container" podUID="3f336286-4447-42e0-98be-cb39b7f0a4ee" containerName="revoke-keys"
Oct 31 13:27:36.828941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:36.828940    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f336286-4447-42e0-98be-cb39b7f0a4ee" containerName="revoke-keys"
Oct 31 13:27:36.829134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:36.829088    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f336286-4447-42e0-98be-cb39b7f0a4ee" containerName="revoke-keys"
Oct 31 13:27:36.834097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:36.834076    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:36.840384 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:36.840053    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aabeef\"/\"distinct-2-w6-aabeef-kserve-self-signed-certs\""
Oct 31 13:27:36.844382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:36.844357    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"]
Oct 31 13:27:36.938556 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:36.938528    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:36.985649 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:36.985551    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-model-cache\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:36.986963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:36.986931    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b950b997-7653-480d-b120-31dc384832d0-tls-certs\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:36.987128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:36.986997    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-home\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:36.987195 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:36.987126    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcbc9\" (UniqueName: \"kubernetes.io/projected/b950b997-7653-480d-b120-31dc384832d0-kube-api-access-rcbc9\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:36.987195 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:36.987174    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-dshm\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:36.987301 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:36.987217    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-tmp-dir\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:37.088606 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:37.088563    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-home\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:37.088781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:37.088726    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcbc9\" (UniqueName: \"kubernetes.io/projected/b950b997-7653-480d-b120-31dc384832d0-kube-api-access-rcbc9\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:37.088860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:37.088824    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-dshm\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:37.088987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:37.088912    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-tmp-dir\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:37.089052 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:37.089037    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-home\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:37.089097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:37.089053    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-model-cache\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:37.089197 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:37.089175    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b950b997-7653-480d-b120-31dc384832d0-tls-certs\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:37.089475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:37.089454    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-tmp-dir\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:37.089813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:37.089789    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-model-cache\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:37.092312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:37.092292    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-dshm\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:37.099643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:37.099620    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b950b997-7653-480d-b120-31dc384832d0-tls-certs\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:37.108983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:37.108966    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcbc9\" (UniqueName: \"kubernetes.io/projected/b950b997-7653-480d-b120-31dc384832d0-kube-api-access-rcbc9\") pod \"distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") " pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:37.163361 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:37.163338    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:37.454850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:37.454817    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"]
Oct 31 13:27:37.455006 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:27:37.454823    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb950b997_7653_480d_b120_31dc384832d0.slice/crio-7e4a3d8e1e53fbd4c81ff4a6810e84f22c2722cd9afdd3c8c53790e012ad8d84 WatchSource:0}: Error finding container 7e4a3d8e1e53fbd4c81ff4a6810e84f22c2722cd9afdd3c8c53790e012ad8d84: Status 404 returned error can't find the container with id 7e4a3d8e1e53fbd4c81ff4a6810e84f22c2722cd9afdd3c8c53790e012ad8d84
Oct 31 13:27:37.710947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:37.701502    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:38.310939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:38.310903    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" event={"ID":"b950b997-7653-480d-b120-31dc384832d0","Type":"ContainerStarted","Data":"2cf5db9e407957590da9856d40402d7f20a7c9b83d442497fcee1d7912473abb"}
Oct 31 13:27:38.311346 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:38.310953    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" event={"ID":"b950b997-7653-480d-b120-31dc384832d0","Type":"ContainerStarted","Data":"7e4a3d8e1e53fbd4c81ff4a6810e84f22c2722cd9afdd3c8c53790e012ad8d84"}
Oct 31 13:27:38.333028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:38.332984    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podStartSLOduration=2.332971587 podStartE2EDuration="2.332971587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:38.331437196 +0000 UTC m=+1455.729785824" watchObservedRunningTime="2026-10-31 13:27:38.332971587 +0000 UTC m=+1455.731320212"
Oct 31 13:27:38.521448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:38.520952    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:38.764153 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:38.764122    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"]
Oct 31 13:27:38.767984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:38.767969    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"
Oct 31 13:27:38.774747 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:38.774724    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:27:38.774976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:38.774945    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:27:38.778633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:38.778611    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"]
Oct 31 13:27:38.920843 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:38.920808    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414a88f7-a4a5-45fa-a922-46987a48d81c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3207b368-x89c8\" (UID: \"414a88f7-a4a5-45fa-a922-46987a48d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"
Oct 31 13:27:38.921019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:38.920963    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4s5n\" (UniqueName: \"kubernetes.io/projected/414a88f7-a4a5-45fa-a922-46987a48d81c-kube-api-access-g4s5n\") pod \"maas-api-revoke-sub-3207b368-x89c8\" (UID: \"414a88f7-a4a5-45fa-a922-46987a48d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"
Oct 31 13:27:38.921174 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:38.921155    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414a88f7-a4a5-45fa-a922-46987a48d81c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3207b368-x89c8\" (UID: \"414a88f7-a4a5-45fa-a922-46987a48d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"
Oct 31 13:27:39.022512 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:39.022439    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414a88f7-a4a5-45fa-a922-46987a48d81c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3207b368-x89c8\" (UID: \"414a88f7-a4a5-45fa-a922-46987a48d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"
Oct 31 13:27:39.022512 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:39.022496    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4s5n\" (UniqueName: \"kubernetes.io/projected/414a88f7-a4a5-45fa-a922-46987a48d81c-kube-api-access-g4s5n\") pod \"maas-api-revoke-sub-3207b368-x89c8\" (UID: \"414a88f7-a4a5-45fa-a922-46987a48d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"
Oct 31 13:27:39.022728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:39.022560    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414a88f7-a4a5-45fa-a922-46987a48d81c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3207b368-x89c8\" (UID: \"414a88f7-a4a5-45fa-a922-46987a48d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"
Oct 31 13:27:39.023183 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:39.023155    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414a88f7-a4a5-45fa-a922-46987a48d81c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3207b368-x89c8\" (UID: \"414a88f7-a4a5-45fa-a922-46987a48d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"
Oct 31 13:27:39.025005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:39.024987    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414a88f7-a4a5-45fa-a922-46987a48d81c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3207b368-x89c8\" (UID: \"414a88f7-a4a5-45fa-a922-46987a48d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"
Oct 31 13:27:39.035514 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:39.035490    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4s5n\" (UniqueName: \"kubernetes.io/projected/414a88f7-a4a5-45fa-a922-46987a48d81c-kube-api-access-g4s5n\") pod \"maas-api-revoke-sub-3207b368-x89c8\" (UID: \"414a88f7-a4a5-45fa-a922-46987a48d81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"
Oct 31 13:27:39.083590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:39.083565    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"
Oct 31 13:27:39.163920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:39.163894    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:39.432951 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:27:39.430193    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414a88f7_a4a5_45fa_a922_46987a48d81c.slice/crio-f715ae1d9ed78a8131a6e6e08c06d175d975751b5a7ccca748eb136a351fd9e3 WatchSource:0}: Error finding container f715ae1d9ed78a8131a6e6e08c06d175d975751b5a7ccca748eb136a351fd9e3: Status 404 returned error can't find the container with id f715ae1d9ed78a8131a6e6e08c06d175d975751b5a7ccca748eb136a351fd9e3
Oct 31 13:27:39.434284 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:39.434257    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"]
Oct 31 13:27:39.444320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:39.444270    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:40.226527 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:40.217645    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:40.324675 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:40.324639    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8" event={"ID":"414a88f7-a4a5-45fa-a922-46987a48d81c","Type":"ContainerStarted","Data":"6633f9fbfd0ce91bb86582998aad1a3b968ff55425353561cea50fff324318e6"}
Oct 31 13:27:40.324848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:40.324707    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8" event={"ID":"414a88f7-a4a5-45fa-a922-46987a48d81c","Type":"ContainerStarted","Data":"f715ae1d9ed78a8131a6e6e08c06d175d975751b5a7ccca748eb136a351fd9e3"}
Oct 31 13:27:40.347340 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:40.347303    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8" podStartSLOduration=2.347289659 podStartE2EDuration="2.347289659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:40.344630203 +0000 UTC m=+1457.742978825" watchObservedRunningTime="2026-10-31 13:27:40.347289659 +0000 UTC m=+1457.745638286"
Oct 31 13:27:41.198510 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:41.198414    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:41.333174 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:41.333141    4428 generic.go:356] "Generic (PLEG): container finished" podID="414a88f7-a4a5-45fa-a922-46987a48d81c" containerID="6633f9fbfd0ce91bb86582998aad1a3b968ff55425353561cea50fff324318e6" exitCode=0
Oct 31 13:27:41.333368 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:41.333229    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8" event={"ID":"414a88f7-a4a5-45fa-a922-46987a48d81c","Type":"ContainerDied","Data":"6633f9fbfd0ce91bb86582998aad1a3b968ff55425353561cea50fff324318e6"}
Oct 31 13:27:41.494583 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:41.494550    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:42.130935 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:42.127730    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:42.531194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:42.531173    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"
Oct 31 13:27:42.676711 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:42.676640    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414a88f7-a4a5-45fa-a922-46987a48d81c-kube-api-access-g4s5n\" (UniqueName: \"kubernetes.io/projected/414a88f7-a4a5-45fa-a922-46987a48d81c-kube-api-access-g4s5n\") pod \"414a88f7-a4a5-45fa-a922-46987a48d81c\" (UID: \"414a88f7-a4a5-45fa-a922-46987a48d81c\") "
Oct 31 13:27:42.676842 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:42.676712    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/414a88f7-a4a5-45fa-a922-46987a48d81c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414a88f7-a4a5-45fa-a922-46987a48d81c-maas-api-xks-ca\") pod \"414a88f7-a4a5-45fa-a922-46987a48d81c\" (UID: \"414a88f7-a4a5-45fa-a922-46987a48d81c\") "
Oct 31 13:27:42.676842 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:42.676782    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414a88f7-a4a5-45fa-a922-46987a48d81c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414a88f7-a4a5-45fa-a922-46987a48d81c-maas-api-openshift-ca\") pod \"414a88f7-a4a5-45fa-a922-46987a48d81c\" (UID: \"414a88f7-a4a5-45fa-a922-46987a48d81c\") "
Oct 31 13:27:42.677212 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:42.677189    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414a88f7-a4a5-45fa-a922-46987a48d81c-maas-api-openshift-ca" pod "414a88f7-a4a5-45fa-a922-46987a48d81c" (UID: "414a88f7-a4a5-45fa-a922-46987a48d81c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:27:42.679077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:42.679053    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414a88f7-a4a5-45fa-a922-46987a48d81c-kube-api-access-g4s5n" pod "414a88f7-a4a5-45fa-a922-46987a48d81c" (UID: "414a88f7-a4a5-45fa-a922-46987a48d81c"). InnerVolumeSpecName "kube-api-access-g4s5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:27:42.679178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:42.679074    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/414a88f7-a4a5-45fa-a922-46987a48d81c-maas-api-xks-ca" pod "414a88f7-a4a5-45fa-a922-46987a48d81c" (UID: "414a88f7-a4a5-45fa-a922-46987a48d81c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:27:42.777724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:42.777695    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4s5n\" (UniqueName: \"kubernetes.io/projected/414a88f7-a4a5-45fa-a922-46987a48d81c-kube-api-access-g4s5n\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:42.777724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:42.777722    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414a88f7-a4a5-45fa-a922-46987a48d81c-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:42.777892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:42.777738    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414a88f7-a4a5-45fa-a922-46987a48d81c-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:42.952915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:42.948952    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:43.346025 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:43.345987    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"
Oct 31 13:27:43.346199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:43.346016    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8" event={"ID":"414a88f7-a4a5-45fa-a922-46987a48d81c","Type":"ContainerDied","Data":"f715ae1d9ed78a8131a6e6e08c06d175d975751b5a7ccca748eb136a351fd9e3"}
Oct 31 13:27:43.346199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:43.346080    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f715ae1d9ed78a8131a6e6e08c06d175d975751b5a7ccca748eb136a351fd9e3"
Oct 31 13:27:43.976892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:43.973554    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:44.905034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:44.904999    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:46.763232 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:46.763186    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:47.164354 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:47.164250    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:47.192319 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:47.192288    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:47.397705 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:47.397673    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:27:48.013190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.013105    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:48.089710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.089675    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"]
Oct 31 13:27:48.090227 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.090213    4428 cpu_manager.go:414] "containerMap: removing container" podUID="414a88f7-a4a5-45fa-a922-46987a48d81c" containerName="revoke-keys"
Oct 31 13:27:48.090286 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.090229    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414a88f7-a4a5-45fa-a922-46987a48d81c" containerName="revoke-keys"
Oct 31 13:27:48.090340 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.090331    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="414a88f7-a4a5-45fa-a922-46987a48d81c" containerName="revoke-keys"
Oct 31 13:27:48.093958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.093941    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"
Oct 31 13:27:48.100719 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.100698    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:27:48.100936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.100909    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:27:48.103813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.103790    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"]
Oct 31 13:27:48.178964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.178934    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c98b0b-a08c-453a-a83f-2605f568ef8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c514b01f-slkxh\" (UID: \"79c98b0b-a08c-453a-a83f-2605f568ef8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"
Oct 31 13:27:48.179118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.178997    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdn4b\" (UniqueName: \"kubernetes.io/projected/79c98b0b-a08c-453a-a83f-2605f568ef8d-kube-api-access-pdn4b\") pod \"maas-api-revoke-sub-c514b01f-slkxh\" (UID: \"79c98b0b-a08c-453a-a83f-2605f568ef8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"
Oct 31 13:27:48.179118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.179078    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c98b0b-a08c-453a-a83f-2605f568ef8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c514b01f-slkxh\" (UID: \"79c98b0b-a08c-453a-a83f-2605f568ef8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"
Oct 31 13:27:48.280395 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.280294    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c98b0b-a08c-453a-a83f-2605f568ef8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c514b01f-slkxh\" (UID: \"79c98b0b-a08c-453a-a83f-2605f568ef8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"
Oct 31 13:27:48.280395 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.280383    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdn4b\" (UniqueName: \"kubernetes.io/projected/79c98b0b-a08c-453a-a83f-2605f568ef8d-kube-api-access-pdn4b\") pod \"maas-api-revoke-sub-c514b01f-slkxh\" (UID: \"79c98b0b-a08c-453a-a83f-2605f568ef8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"
Oct 31 13:27:48.280623 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.280425    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c98b0b-a08c-453a-a83f-2605f568ef8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c514b01f-slkxh\" (UID: \"79c98b0b-a08c-453a-a83f-2605f568ef8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"
Oct 31 13:27:48.280965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.280943    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c98b0b-a08c-453a-a83f-2605f568ef8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c514b01f-slkxh\" (UID: \"79c98b0b-a08c-453a-a83f-2605f568ef8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"
Oct 31 13:27:48.282959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.282941    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c98b0b-a08c-453a-a83f-2605f568ef8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c514b01f-slkxh\" (UID: \"79c98b0b-a08c-453a-a83f-2605f568ef8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"
Oct 31 13:27:48.293490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.293465    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdn4b\" (UniqueName: \"kubernetes.io/projected/79c98b0b-a08c-453a-a83f-2605f568ef8d-kube-api-access-pdn4b\") pod \"maas-api-revoke-sub-c514b01f-slkxh\" (UID: \"79c98b0b-a08c-453a-a83f-2605f568ef8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"
Oct 31 13:27:48.410716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.410694    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"
Oct 31 13:27:48.699523 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:48.699496    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"]
Oct 31 13:27:48.700728 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:27:48.700706    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c98b0b_a08c_453a_a83f_2605f568ef8d.slice/crio-b760a5303f467d458a5570cbd6e558d3eba2ef65ca1d1b680b49d2b86804b81a WatchSource:0}: Error finding container b760a5303f467d458a5570cbd6e558d3eba2ef65ca1d1b680b49d2b86804b81a: Status 404 returned error can't find the container with id b760a5303f467d458a5570cbd6e558d3eba2ef65ca1d1b680b49d2b86804b81a
Oct 31 13:27:48.739850 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:27:48.739825    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:27:49.036369 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:49.036336    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:49.395373 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:49.395284    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh" event={"ID":"79c98b0b-a08c-453a-a83f-2605f568ef8d","Type":"ContainerStarted","Data":"36f45770657d8925f2985f2ac47a64bcc37918564316e767dc0aee131033f039"}
Oct 31 13:27:49.395578 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:49.395453    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh" event={"ID":"79c98b0b-a08c-453a-a83f-2605f568ef8d","Type":"ContainerStarted","Data":"b760a5303f467d458a5570cbd6e558d3eba2ef65ca1d1b680b49d2b86804b81a"}
Oct 31 13:27:49.418385 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:49.418331    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh" podStartSLOduration=1.4183172960000001 podStartE2EDuration="1.418317296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:49.416949694 +0000 UTC m=+1466.815298319" watchObservedRunningTime="2026-10-31 13:27:49.418317296 +0000 UTC m=+1466.816665919"
Oct 31 13:27:49.740762 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:49.740707    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:49.949818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:49.949783    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:50.403137 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:50.403106    4428 generic.go:356] "Generic (PLEG): container finished" podID="79c98b0b-a08c-453a-a83f-2605f568ef8d" containerID="36f45770657d8925f2985f2ac47a64bcc37918564316e767dc0aee131033f039" exitCode=0
Oct 31 13:27:50.403495 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:50.403155    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh" event={"ID":"79c98b0b-a08c-453a-a83f-2605f568ef8d","Type":"ContainerDied","Data":"36f45770657d8925f2985f2ac47a64bcc37918564316e767dc0aee131033f039"}
Oct 31 13:27:51.101685 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:51.101620    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:51.545986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:51.545965    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"
Oct 31 13:27:51.630187 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:51.630164    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79c98b0b-a08c-453a-a83f-2605f568ef8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c98b0b-a08c-453a-a83f-2605f568ef8d-maas-api-xks-ca\") pod \"79c98b0b-a08c-453a-a83f-2605f568ef8d\" (UID: \"79c98b0b-a08c-453a-a83f-2605f568ef8d\") "
Oct 31 13:27:51.630349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:51.630264    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c98b0b-a08c-453a-a83f-2605f568ef8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c98b0b-a08c-453a-a83f-2605f568ef8d-maas-api-openshift-ca\") pod \"79c98b0b-a08c-453a-a83f-2605f568ef8d\" (UID: \"79c98b0b-a08c-453a-a83f-2605f568ef8d\") "
Oct 31 13:27:51.630417 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:51.630348    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c98b0b-a08c-453a-a83f-2605f568ef8d-kube-api-access-pdn4b\" (UniqueName: \"kubernetes.io/projected/79c98b0b-a08c-453a-a83f-2605f568ef8d-kube-api-access-pdn4b\") pod \"79c98b0b-a08c-453a-a83f-2605f568ef8d\" (UID: \"79c98b0b-a08c-453a-a83f-2605f568ef8d\") "
Oct 31 13:27:51.630651 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:51.630624    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c98b0b-a08c-453a-a83f-2605f568ef8d-maas-api-openshift-ca" pod "79c98b0b-a08c-453a-a83f-2605f568ef8d" (UID: "79c98b0b-a08c-453a-a83f-2605f568ef8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:27:51.630803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:51.630786    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c98b0b-a08c-453a-a83f-2605f568ef8d-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:51.632357 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:51.632330    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c98b0b-a08c-453a-a83f-2605f568ef8d-maas-api-xks-ca" pod "79c98b0b-a08c-453a-a83f-2605f568ef8d" (UID: "79c98b0b-a08c-453a-a83f-2605f568ef8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:27:51.632463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:51.632443    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c98b0b-a08c-453a-a83f-2605f568ef8d-kube-api-access-pdn4b" pod "79c98b0b-a08c-453a-a83f-2605f568ef8d" (UID: "79c98b0b-a08c-453a-a83f-2605f568ef8d"). InnerVolumeSpecName "kube-api-access-pdn4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:27:51.732183 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:51.732153    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c98b0b-a08c-453a-a83f-2605f568ef8d-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:51.732183 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:51.732178    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdn4b\" (UniqueName: \"kubernetes.io/projected/79c98b0b-a08c-453a-a83f-2605f568ef8d-kube-api-access-pdn4b\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:51.997136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:51.997021    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:52.416617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:52.416535    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"
Oct 31 13:27:52.416617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:52.416550    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh" event={"ID":"79c98b0b-a08c-453a-a83f-2605f568ef8d","Type":"ContainerDied","Data":"b760a5303f467d458a5570cbd6e558d3eba2ef65ca1d1b680b49d2b86804b81a"}
Oct 31 13:27:52.416617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:52.416598    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b760a5303f467d458a5570cbd6e558d3eba2ef65ca1d1b680b49d2b86804b81a"
Oct 31 13:27:52.995370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:52.994353    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:53.795640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:53.795594    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:54.612544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:54.612506    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:55.538855 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:55.536285    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:56.328582 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.328552    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"]
Oct 31 13:27:56.329079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.329066    4428 cpu_manager.go:414] "containerMap: removing container" podUID="79c98b0b-a08c-453a-a83f-2605f568ef8d" containerName="revoke-keys"
Oct 31 13:27:56.329125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.329080    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c98b0b-a08c-453a-a83f-2605f568ef8d" containerName="revoke-keys"
Oct 31 13:27:56.329198 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.329188    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c98b0b-a08c-453a-a83f-2605f568ef8d" containerName="revoke-keys"
Oct 31 13:27:56.332731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.332705    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.340894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.340825    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6784d6\"/\"default-dockercfg-6gpxf\""
Oct 31 13:27:56.343083 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.343058    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6784d6\"/\"kube-root-ca.crt\""
Oct 31 13:27:56.343249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.343106    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6784d6\"/\"openshift-service-ca.crt\""
Oct 31 13:27:56.343857 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.343824    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6784d6\"/\"distinct-2-w5-6784d6-kserve-self-signed-certs\""
Oct 31 13:27:56.345544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.345527    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"]
Oct 31 13:27:56.376754 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.376727    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-dshm\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.376929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.376913    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-tmp-dir\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.376979 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.376940    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-model-cache\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.377024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.376974    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlqpw\" (UniqueName: \"kubernetes.io/projected/a85ed201-6d93-4c52-9315-8c733ea37b89-kube-api-access-rlqpw\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.377179 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.377158    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85ed201-6d93-4c52-9315-8c733ea37b89-tls-certs\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.377236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.377197    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-home\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.379532 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.379511    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:56.505891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.489575    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-tmp-dir\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.505891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.489642    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-model-cache\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.505891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.489685    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlqpw\" (UniqueName: \"kubernetes.io/projected/a85ed201-6d93-4c52-9315-8c733ea37b89-kube-api-access-rlqpw\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.505891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.489764    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85ed201-6d93-4c52-9315-8c733ea37b89-tls-certs\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.505891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.489803    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-home\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.505891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.489908    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-dshm\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.505891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.491149    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-tmp-dir\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.505891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.491392    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-model-cache\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.505891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.493162    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-home\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.505891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.501501    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-dshm\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.505891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.502635    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85ed201-6d93-4c52-9315-8c733ea37b89-tls-certs\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.542893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.535542    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlqpw\" (UniqueName: \"kubernetes.io/projected/a85ed201-6d93-4c52-9315-8c733ea37b89-kube-api-access-rlqpw\") pod \"distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") " pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.557892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.551564    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:56.649115 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.649042    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:56.933147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:56.933123    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"]
Oct 31 13:27:56.933757 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:27:56.933730    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda85ed201_6d93_4c52_9315_8c733ea37b89.slice/crio-bc10391b2b35dc5aa3e62d0b675a55c5fada486c0761723b5b10feb8281c60f3 WatchSource:0}: Error finding container bc10391b2b35dc5aa3e62d0b675a55c5fada486c0761723b5b10feb8281c60f3: Status 404 returned error can't find the container with id bc10391b2b35dc5aa3e62d0b675a55c5fada486c0761723b5b10feb8281c60f3
Oct 31 13:27:57.229120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:57.225172    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:57.450424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:57.450385    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" event={"ID":"a85ed201-6d93-4c52-9315-8c733ea37b89","Type":"ContainerStarted","Data":"d42338fd2971a02ebeff8dc4437b59667789764fc75047b524b9fc622d58594f"}
Oct 31 13:27:57.450823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:57.450437    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" event={"ID":"a85ed201-6d93-4c52-9315-8c733ea37b89","Type":"ContainerStarted","Data":"bc10391b2b35dc5aa3e62d0b675a55c5fada486c0761723b5b10feb8281c60f3"}
Oct 31 13:27:57.471984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:57.471943    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podStartSLOduration=1.471930081 podStartE2EDuration="1.471930081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:57.469759821 +0000 UTC m=+1474.868108445" watchObservedRunningTime="2026-10-31 13:27:57.471930081 +0000 UTC m=+1474.870278704"
Oct 31 13:27:57.649439 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:57.649369    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:27:58.069822 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.069789    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:58.351809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.351737    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"]
Oct 31 13:27:58.355781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.355762    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.365471 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.365139    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"]
Oct 31 13:27:58.427673 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.427646    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.427821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.427761    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93314798-1f48-408a-b87c-d8f809d2f14b-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.427915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.427832    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93314798-1f48-408a-b87c-d8f809d2f14b-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.427968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.427940    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.427968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.427958    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.428036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.428003    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6qj9\" (UniqueName: \"kubernetes.io/projected/93314798-1f48-408a-b87c-d8f809d2f14b-kube-api-access-b6qj9\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.428036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.428026    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.428132 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.428094    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93314798-1f48-408a-b87c-d8f809d2f14b-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.428168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.428146    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.529202 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.529126    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.529202 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.529165    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.529738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.529221    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6qj9\" (UniqueName: \"kubernetes.io/projected/93314798-1f48-408a-b87c-d8f809d2f14b-kube-api-access-b6qj9\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.529738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.529250    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.529738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.529275    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93314798-1f48-408a-b87c-d8f809d2f14b-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.529738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.529315    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.529738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.529349    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.529738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.529506    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93314798-1f48-408a-b87c-d8f809d2f14b-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.529738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.529557    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.529738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.529568    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93314798-1f48-408a-b87c-d8f809d2f14b-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.529738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.529590    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.530391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.529783    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.530391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.529981    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.530556 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.530530    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93314798-1f48-408a-b87c-d8f809d2f14b-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.533044 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.533017    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.533318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.533299    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93314798-1f48-408a-b87c-d8f809d2f14b-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.544500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.544478    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93314798-1f48-408a-b87c-d8f809d2f14b-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.545057 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.545028    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6qj9\" (UniqueName: \"kubernetes.io/projected/93314798-1f48-408a-b87c-d8f809d2f14b-kube-api-access-b6qj9\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.672274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.672192    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:58.825892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.825297    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:27:58.881306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.881276    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"]
Oct 31 13:27:58.886145 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.886124    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:58.903328 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.903296    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d85d82-openshift-default-dockercfg-xn9td\""
Oct 31 13:27:58.906584 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.906564    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"]
Oct 31 13:27:58.958772 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.958738    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"]
Oct 31 13:27:58.960561 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:27:58.960504    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93314798_1f48_408a_b87c_d8f809d2f14b.slice/crio-6af5a27412fa8cd8a089c7a6144d6659f134566beb1f392c1b0d9162a29db245 WatchSource:0}: Error finding container 6af5a27412fa8cd8a089c7a6144d6659f134566beb1f392c1b0d9162a29db245: Status 404 returned error can't find the container with id 6af5a27412fa8cd8a089c7a6144d6659f134566beb1f392c1b0d9162a29db245
Oct 31 13:27:58.963008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.962981    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:27:58.963072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.963038    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:27:58.963072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:58.963064    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:27:59.040983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.040956    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.041087 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.041008    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.041157 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.041138    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.041195 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.041168    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.041247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.041238    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.041291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.041262    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m258\" (UniqueName: \"kubernetes.io/projected/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-kube-api-access-8m258\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.041327 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.041316    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.041424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.041410    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.041456 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.041439    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.142752 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.142722    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.142911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.142797    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.142911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.142828    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.142911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.142858    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.143082 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.142912    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.143082 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.142983    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.143082 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.143008    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.143082 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.143071    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.143280 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.143106    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m258\" (UniqueName: \"kubernetes.io/projected/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-kube-api-access-8m258\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.143493 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.143445    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.143493 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.143483    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.143852 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.143827    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.144209 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.144187    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.144290 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.144188    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.145761 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.145741    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.145848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.145780    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.163357 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.163332    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.171525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.171501    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m258\" (UniqueName: \"kubernetes.io/projected/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-kube-api-access-8m258\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.200511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.200488    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:27:59.463811 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.463728    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns" event={"ID":"93314798-1f48-408a-b87c-d8f809d2f14b","Type":"ContainerStarted","Data":"efa22e7ed2f6b7bf67c47aba76e075d62775b017722de4af5944aa9e1545e58f"}
Oct 31 13:27:59.463811 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.463797    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns" event={"ID":"93314798-1f48-408a-b87c-d8f809d2f14b","Type":"ContainerStarted","Data":"6af5a27412fa8cd8a089c7a6144d6659f134566beb1f392c1b0d9162a29db245"}
Oct 31 13:27:59.489450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.489424    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"]
Oct 31 13:27:59.490607 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:27:59.490582    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b5a82a_0de7_4c45_94a5_b8c046ea9387.slice/crio-93e24e46a69f157d347908b6b0b6d643a5ec5be814ee45cbd214a4474eb87129 WatchSource:0}: Error finding container 93e24e46a69f157d347908b6b0b6d643a5ec5be814ee45cbd214a4474eb87129: Status 404 returned error can't find the container with id 93e24e46a69f157d347908b6b0b6d643a5ec5be814ee45cbd214a4474eb87129
Oct 31 13:27:59.491309 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.491269    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns" podStartSLOduration=1.4912555699999999 podStartE2EDuration="1.49125557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:59.489664904 +0000 UTC m=+1476.888013529" watchObservedRunningTime="2026-10-31 13:27:59.49125557 +0000 UTC m=+1476.889604263"
Oct 31 13:27:59.493323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.493290    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:27:59.493405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.493364    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:27:59.493458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.493404    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:27:59.672984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.672951    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:27:59.802958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:27:59.800991    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:00.497091 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:00.496957    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst" event={"ID":"f5b5a82a-0de7-4c45-94a5-b8c046ea9387","Type":"ContainerStarted","Data":"f161292e68d5d1ccd5e62af7417e5a70b5171e7af061eb453aee43bda43a9f62"}
Oct 31 13:28:00.497091 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:00.497051    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst" event={"ID":"f5b5a82a-0de7-4c45-94a5-b8c046ea9387","Type":"ContainerStarted","Data":"93e24e46a69f157d347908b6b0b6d643a5ec5be814ee45cbd214a4474eb87129"}
Oct 31 13:28:00.525649 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:00.525598    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst" podStartSLOduration=2.525584181 podStartE2EDuration="2.525584181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:00.522918891 +0000 UTC m=+1477.921267516" watchObservedRunningTime="2026-10-31 13:28:00.525584181 +0000 UTC m=+1477.923932804"
Oct 31 13:28:00.685019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:00.684982    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 13:28:00.685618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:00.685591    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns" podUID="93314798-1f48-408a-b87c-d8f809d2f14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 13:28:00.725495 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:00.725456    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:01.202009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:01.201979    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:28:01.202735 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:01.202713    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 13:28:01.202783 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:01.202763    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst" podUID="f5b5a82a-0de7-4c45-94a5-b8c046ea9387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 13:28:01.682893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:01.682268    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 13:28:01.682893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:01.682325    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns" podUID="93314798-1f48-408a-b87c-d8f809d2f14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 13:28:01.930894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:01.910820    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:02.201516 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:02.201479    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 13:28:02.201678 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:02.201543    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst" podUID="f5b5a82a-0de7-4c45-94a5-b8c046ea9387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 13:28:02.676894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:02.676194    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 13:28:02.676894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:02.676262    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns" podUID="93314798-1f48-408a-b87c-d8f809d2f14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 13:28:02.897891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:02.897662    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:03.201598 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:03.201564    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 13:28:03.201987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:03.201648    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst" podUID="f5b5a82a-0de7-4c45-94a5-b8c046ea9387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 13:28:03.681374 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:03.681340    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 13:28:03.681697 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:03.681647    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns" podUID="93314798-1f48-408a-b87c-d8f809d2f14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 13:28:03.745515 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:03.745427    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:04.201491 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:04.201405    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 13:28:04.201491 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:04.201456    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst" podUID="f5b5a82a-0de7-4c45-94a5-b8c046ea9387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 13:28:04.652993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:04.651566    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:04.675894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:04.673256    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 13:28:04.675894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:04.673327    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns" podUID="93314798-1f48-408a-b87c-d8f809d2f14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 13:28:05.201534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:05.201485    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 13:28:05.201756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:05.201542    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst" podUID="f5b5a82a-0de7-4c45-94a5-b8c046ea9387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 13:28:05.673076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:05.673034    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 13:28:05.673255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:05.673117    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns" podUID="93314798-1f48-408a-b87c-d8f809d2f14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 13:28:05.809152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:05.809113    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:06.163441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:06.163135    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:06.204249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:06.203713    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 13:28:06.204249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:06.203784    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst" podUID="f5b5a82a-0de7-4c45-94a5-b8c046ea9387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 13:28:06.667411 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:06.660410    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:28:06.702903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:06.701743    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:28:07.297037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:07.296998    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:07.688554 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:07.687942    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:28:07.688554 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:07.688010    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns" podUID="93314798-1f48-408a-b87c-d8f809d2f14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:07.780894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:07.780173    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:28:08.196755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:08.196714    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:28:08.198714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:08.198696    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:28:08.210347 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:08.210325    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:28:08.566890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:08.562240    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:08.626198 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:08.626169    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:28:08.627797 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:08.627774    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:28:08.629415 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:08.629395    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:28:09.536840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:09.536807    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:09.702256 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:09.702227    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:10.528899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:10.526703    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:11.400144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:11.400109    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:12.372441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:12.372380    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:12.491696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:12.491661    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"]
Oct 31 13:28:12.495769 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:12.495748    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"
Oct 31 13:28:12.507899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:12.506041    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:28:12.507899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:12.506101    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:28:12.513382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:12.511169    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"]
Oct 31 13:28:12.533053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:12.533023    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:12.632835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:12.632758    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d97567d5-0707-423e-bb4c-213ceb96f80e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69d68719-l2k98\" (UID: \"d97567d5-0707-423e-bb4c-213ceb96f80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"
Oct 31 13:28:12.632997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:12.632903    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d97567d5-0707-423e-bb4c-213ceb96f80e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69d68719-l2k98\" (UID: \"d97567d5-0707-423e-bb4c-213ceb96f80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"
Oct 31 13:28:12.632997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:12.632964    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjvrg\" (UniqueName: \"kubernetes.io/projected/d97567d5-0707-423e-bb4c-213ceb96f80e-kube-api-access-sjvrg\") pod \"maas-api-revoke-sub-69d68719-l2k98\" (UID: \"d97567d5-0707-423e-bb4c-213ceb96f80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"
Oct 31 13:28:12.733484 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:12.733457    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjvrg\" (UniqueName: \"kubernetes.io/projected/d97567d5-0707-423e-bb4c-213ceb96f80e-kube-api-access-sjvrg\") pod \"maas-api-revoke-sub-69d68719-l2k98\" (UID: \"d97567d5-0707-423e-bb4c-213ceb96f80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"
Oct 31 13:28:12.733594 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:12.733552    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d97567d5-0707-423e-bb4c-213ceb96f80e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69d68719-l2k98\" (UID: \"d97567d5-0707-423e-bb4c-213ceb96f80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"
Oct 31 13:28:12.733659 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:12.733621    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d97567d5-0707-423e-bb4c-213ceb96f80e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69d68719-l2k98\" (UID: \"d97567d5-0707-423e-bb4c-213ceb96f80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"
Oct 31 13:28:12.734306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:12.734284    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d97567d5-0707-423e-bb4c-213ceb96f80e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69d68719-l2k98\" (UID: \"d97567d5-0707-423e-bb4c-213ceb96f80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"
Oct 31 13:28:12.736227 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:12.736203    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d97567d5-0707-423e-bb4c-213ceb96f80e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69d68719-l2k98\" (UID: \"d97567d5-0707-423e-bb4c-213ceb96f80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"
Oct 31 13:28:12.752094 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:12.752075    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjvrg\" (UniqueName: \"kubernetes.io/projected/d97567d5-0707-423e-bb4c-213ceb96f80e-kube-api-access-sjvrg\") pod \"maas-api-revoke-sub-69d68719-l2k98\" (UID: \"d97567d5-0707-423e-bb4c-213ceb96f80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"
Oct 31 13:28:12.816587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:12.816563    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"
Oct 31 13:28:13.101798 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.101769    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"]
Oct 31 13:28:13.102642 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:28:13.102620    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97567d5_0707_423e_bb4c_213ceb96f80e.slice/crio-f87d2fd10d86877ef7bccd8192f141af780f85bdfcce010d8cf712b45fb14d7c WatchSource:0}: Error finding container f87d2fd10d86877ef7bccd8192f141af780f85bdfcce010d8cf712b45fb14d7c: Status 404 returned error can't find the container with id f87d2fd10d86877ef7bccd8192f141af780f85bdfcce010d8cf712b45fb14d7c
Oct 31 13:28:13.319893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.314316    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:13.618981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.618903    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98" event={"ID":"d97567d5-0707-423e-bb4c-213ceb96f80e","Type":"ContainerStarted","Data":"e93ced3ac204493d9fc19d879700b414ec5329d250349a22e90fb41b52568a9a"}
Oct 31 13:28:13.618981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.618951    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98" event={"ID":"d97567d5-0707-423e-bb4c-213ceb96f80e","Type":"ContainerStarted","Data":"f87d2fd10d86877ef7bccd8192f141af780f85bdfcce010d8cf712b45fb14d7c"}
Oct 31 13:28:13.639435 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.639387    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98" podStartSLOduration=1.6393695099999999 podStartE2EDuration="1.63936951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:13.637730493 +0000 UTC m=+1491.036079118" watchObservedRunningTime="2026-10-31 13:28:13.63936951 +0000 UTC m=+1491.037718134"
Oct 31 13:28:13.756702 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.756676    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"]
Oct 31 13:28:13.760720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.760701    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.777929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.777906    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"]
Oct 31 13:28:13.845991 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.845957    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.846150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.845999    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.846150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.846133    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.846223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.846173    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.846301 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.846269    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.846379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.846360    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.846468 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.846451    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.846528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.846516    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.846618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.846604    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg9jt\" (UniqueName: \"kubernetes.io/projected/aaab76aa-bfe5-4413-83a8-bc1037c2b525-kube-api-access-lg9jt\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.947818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.947788    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.947978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.947848    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.947978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.947935    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.947978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.947968    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg9jt\" (UniqueName: \"kubernetes.io/projected/aaab76aa-bfe5-4413-83a8-bc1037c2b525-kube-api-access-lg9jt\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.948163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.948012    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.948163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.948030    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.948163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.948101    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.948163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.948138    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.948384 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.948188    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.948444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.948393    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.948502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.948473    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.948843 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.948820    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.949162 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.949136    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.949433 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.949410    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.950402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.950382    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.951119 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.951101    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.962191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.962171    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg9jt\" (UniqueName: \"kubernetes.io/projected/aaab76aa-bfe5-4413-83a8-bc1037c2b525-kube-api-access-lg9jt\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:13.962364 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:13.962348    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:14.075125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:14.075098    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:14.262037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:14.262007    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:14.375300 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:14.375276    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"]
Oct 31 13:28:14.376231 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:28:14.376206    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaab76aa_bfe5_4413_83a8_bc1037c2b525.slice/crio-a50c69ec23b1b82efd159356d6c135121f270c5623b213fed235276ebb1f0f84 WatchSource:0}: Error finding container a50c69ec23b1b82efd159356d6c135121f270c5623b213fed235276ebb1f0f84: Status 404 returned error can't find the container with id a50c69ec23b1b82efd159356d6c135121f270c5623b213fed235276ebb1f0f84
Oct 31 13:28:14.378772 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:14.378735    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:14.378917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:14.378830    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:14.378967 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:14.378953    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:14.627763 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:14.627675    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx" event={"ID":"aaab76aa-bfe5-4413-83a8-bc1037c2b525","Type":"ContainerStarted","Data":"bdee988fff66234b73085f7d21cb3c50213f9a09b6e07328bb1720a57d4f3fab"}
Oct 31 13:28:14.627763 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:14.627735    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx" event={"ID":"aaab76aa-bfe5-4413-83a8-bc1037c2b525","Type":"ContainerStarted","Data":"a50c69ec23b1b82efd159356d6c135121f270c5623b213fed235276ebb1f0f84"}
Oct 31 13:28:14.629592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:14.629567    4428 generic.go:356] "Generic (PLEG): container finished" podID="d97567d5-0707-423e-bb4c-213ceb96f80e" containerID="e93ced3ac204493d9fc19d879700b414ec5329d250349a22e90fb41b52568a9a" exitCode=0
Oct 31 13:28:14.629592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:14.629602    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98" event={"ID":"d97567d5-0707-423e-bb4c-213ceb96f80e","Type":"ContainerDied","Data":"e93ced3ac204493d9fc19d879700b414ec5329d250349a22e90fb41b52568a9a"}
Oct 31 13:28:14.653144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:14.651745    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx" podStartSLOduration=1.651731646 podStartE2EDuration="1.651731646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:14.650944618 +0000 UTC m=+1492.049293243" watchObservedRunningTime="2026-10-31 13:28:14.651731646 +0000 UTC m=+1492.050080269"
Oct 31 13:28:15.013902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:15.010825    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:15.076235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:15.076197    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:15.077985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:15.077959    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:28:15.078176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:15.078157    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx" podUID="aaab76aa-bfe5-4413-83a8-bc1037c2b525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 13:28:15.841049 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:15.841023    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"
Oct 31 13:28:15.983895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:15.981281    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d97567d5-0707-423e-bb4c-213ceb96f80e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d97567d5-0707-423e-bb4c-213ceb96f80e-maas-api-xks-ca\") pod \"d97567d5-0707-423e-bb4c-213ceb96f80e\" (UID: \"d97567d5-0707-423e-bb4c-213ceb96f80e\") "
Oct 31 13:28:15.983895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:15.981469    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d97567d5-0707-423e-bb4c-213ceb96f80e-kube-api-access-sjvrg\" (UniqueName: \"kubernetes.io/projected/d97567d5-0707-423e-bb4c-213ceb96f80e-kube-api-access-sjvrg\") pod \"d97567d5-0707-423e-bb4c-213ceb96f80e\" (UID: \"d97567d5-0707-423e-bb4c-213ceb96f80e\") "
Oct 31 13:28:15.983895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:15.981666    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d97567d5-0707-423e-bb4c-213ceb96f80e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d97567d5-0707-423e-bb4c-213ceb96f80e-maas-api-openshift-ca\") pod \"d97567d5-0707-423e-bb4c-213ceb96f80e\" (UID: \"d97567d5-0707-423e-bb4c-213ceb96f80e\") "
Oct 31 13:28:15.983895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:15.983026    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:15.983895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:15.983524    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d97567d5-0707-423e-bb4c-213ceb96f80e-maas-api-openshift-ca" pod "d97567d5-0707-423e-bb4c-213ceb96f80e" (UID: "d97567d5-0707-423e-bb4c-213ceb96f80e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:28:15.986328 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:15.986304    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97567d5-0707-423e-bb4c-213ceb96f80e-kube-api-access-sjvrg" pod "d97567d5-0707-423e-bb4c-213ceb96f80e" (UID: "d97567d5-0707-423e-bb4c-213ceb96f80e"). InnerVolumeSpecName "kube-api-access-sjvrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:28:15.986570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:15.986551    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d97567d5-0707-423e-bb4c-213ceb96f80e-maas-api-xks-ca" pod "d97567d5-0707-423e-bb4c-213ceb96f80e" (UID: "d97567d5-0707-423e-bb4c-213ceb96f80e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:28:16.075604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.075574    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:28:16.075757 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.075620    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx" podUID="aaab76aa-bfe5-4413-83a8-bc1037c2b525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 13:28:16.084209 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.084177    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjvrg\" (UniqueName: \"kubernetes.io/projected/d97567d5-0707-423e-bb4c-213ceb96f80e-kube-api-access-sjvrg\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:16.084352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.084213    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d97567d5-0707-423e-bb4c-213ceb96f80e-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:16.084352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.084231    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d97567d5-0707-423e-bb4c-213ceb96f80e-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:16.642789 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.642760    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"
Oct 31 13:28:16.643017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.642797    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98" event={"ID":"d97567d5-0707-423e-bb4c-213ceb96f80e","Type":"ContainerDied","Data":"f87d2fd10d86877ef7bccd8192f141af780f85bdfcce010d8cf712b45fb14d7c"}
Oct 31 13:28:16.643017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.642885    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f87d2fd10d86877ef7bccd8192f141af780f85bdfcce010d8cf712b45fb14d7c"
Oct 31 13:28:16.717703 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.717666    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:16.925854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.925775    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"]
Oct 31 13:28:16.926455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.926437    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d97567d5-0707-423e-bb4c-213ceb96f80e" containerName="revoke-keys"
Oct 31 13:28:16.926538 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.926455    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d97567d5-0707-423e-bb4c-213ceb96f80e" containerName="revoke-keys"
Oct 31 13:28:16.926621 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.926597    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d97567d5-0707-423e-bb4c-213ceb96f80e" containerName="revoke-keys"
Oct 31 13:28:16.930268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.930241    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:16.936718 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.936683    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d85d82\"/\"default-dockercfg-jzcf5\""
Oct 31 13:28:16.937315 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.936785    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d85d82\"/\"distinct-w1-d85d82-kserve-self-signed-certs\""
Oct 31 13:28:16.941150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.940909    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d85d82\"/\"openshift-service-ca.crt\""
Oct 31 13:28:16.941150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.941079    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d85d82\"/\"kube-root-ca.crt\""
Oct 31 13:28:16.973144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:16.973092    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"]
Oct 31 13:28:17.021766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.021728    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:17.086391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.086356    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:28:17.086728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.086692    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx" podUID="aaab76aa-bfe5-4413-83a8-bc1037c2b525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 13:28:17.099849 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.099801    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-tmp-dir\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.100252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.100223    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-dshm\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.100474 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.100456    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-home\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.100698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.100678    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-model-cache\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.100912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.100894    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-tls-certs\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.101068 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.101051    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvm85\" (UniqueName: \"kubernetes.io/projected/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-kube-api-access-nvm85\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.202686 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.202588    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-tls-certs\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.202686 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.202627    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvm85\" (UniqueName: \"kubernetes.io/projected/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-kube-api-access-nvm85\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.202946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.202703    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-tmp-dir\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.202946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.202725    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-dshm\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.202946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.202763    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-home\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.202946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.202814    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-model-cache\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.203218 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.203196    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-model-cache\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.203614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.203593    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-tmp-dir\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.204313 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.204282    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-home\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.206172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.206145    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-dshm\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.206500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.206477    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-tls-certs\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.217264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.217238    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvm85\" (UniqueName: \"kubernetes.io/projected/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-kube-api-access-nvm85\") pod \"distinct-w1-d85d82-kserve-74d7d4576c-26mln\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") " pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.273118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.273083    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:17.563938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.559760    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:17.604490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.597477    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"]
Oct 31 13:28:17.677421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:17.677383    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" event={"ID":"0aa9e815-d6f7-4c18-810a-a66a0cb86be6","Type":"ContainerStarted","Data":"9cfa3207f47acb660041a3d93ee24486e03cd8c35bc75c542a916adcd11e43d4"}
Oct 31 13:28:18.075406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:18.075372    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:28:18.075800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:18.075420    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx" podUID="aaab76aa-bfe5-4413-83a8-bc1037c2b525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 13:28:18.417982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:18.417390    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:18.689517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:18.689419    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" event={"ID":"0aa9e815-d6f7-4c18-810a-a66a0cb86be6","Type":"ContainerStarted","Data":"f3cf6799dbfcfca846aa864ec3c530e9c7aca394c4b8b745f3cf96a04f8a3cc3"}
Oct 31 13:28:18.712305 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:18.712262    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podStartSLOduration=2.7122494379999997 podStartE2EDuration="2.712249438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:18.711696569 +0000 UTC m=+1496.110045195" watchObservedRunningTime="2026-10-31 13:28:18.712249438 +0000 UTC m=+1496.110598062"
Oct 31 13:28:18.740991 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:28:18.740956    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:28:19.076286 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:19.076246    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:28:19.076679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:19.076300    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx" podUID="aaab76aa-bfe5-4413-83a8-bc1037c2b525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 13:28:19.275451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:19.275415    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:19.372660 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:19.372566    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:20.085900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:20.082265    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:28:20.085900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:20.082324    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx" podUID="aaab76aa-bfe5-4413-83a8-bc1037c2b525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 13:28:20.099159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:20.099128    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:20.522515 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:20.522462    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:21.075762 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:21.075728    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:28:21.075962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:21.075802    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx" podUID="aaab76aa-bfe5-4413-83a8-bc1037c2b525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 13:28:21.240309 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:21.240271    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:22.002893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:22.001984    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:22.080092 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:22.080063    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:22.080558 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:22.080537    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:22.081193 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:22.081177    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:28:22.924893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:22.924492    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:23.708887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:23.708832    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:24.482031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:24.481997    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:25.510661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:25.510621    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:26.135169 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:26.135137    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:26.426942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:26.426098    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:27.274235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:27.274201    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:27.286592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:27.286564    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:27.561851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:27.561761    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:27.792185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:27.792150    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:29.085894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.085574    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"]
Oct 31 13:28:29.169893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.169414    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"]
Oct 31 13:28:29.232902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.215561    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.431952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.358024    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.431952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.423661    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.431952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.423702    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.431952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.423812    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.431952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.423847    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.434108 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.434076    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8ggx\" (UniqueName: \"kubernetes.io/projected/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-kube-api-access-q8ggx\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.448090 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.448055    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.448387 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.448365    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.448538 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.448520    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.466894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.462964    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:29.472968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.472931    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:29.478901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.478493    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"]
Oct 31 13:28:29.536020 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.535974    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:29.579894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.550596    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.579894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.550681    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.579894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.550717    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.579894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.550806    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.579894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.550835    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.579894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.550986    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8ggx\" (UniqueName: \"kubernetes.io/projected/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-kube-api-access-q8ggx\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.579894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.551065    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.579894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.551147    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.579894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.551189    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.579894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.551797    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.589900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.586552    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.589900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.586906    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.589900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.587542    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.606664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.606631    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.628323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.607545    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:29.628644 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.611375    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.634905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.612496    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.677919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.677886    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.692000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.690379    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.734255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.725916    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"]
Oct 31 13:28:29.796723 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.796685    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8ggx\" (UniqueName: \"kubernetes.io/projected/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-kube-api-access-q8ggx\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.803417 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.803189    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:29.821262 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.820015    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:28:29.839992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.839959    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.840310 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.840285    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.854048 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.850106    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.854048 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.850151    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.854048 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.850264    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.854048 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.850358    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.854048 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.850435    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84bf9\" (UniqueName: \"kubernetes.io/projected/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-kube-api-access-84bf9\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.854048 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.850506    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.854048 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.850589    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.940894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.940770    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:29.966001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.955247    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.966001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.955296    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.966001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.955330    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.966001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.955411    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.966001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.955480    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.966001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.955546    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84bf9\" (UniqueName: \"kubernetes.io/projected/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-kube-api-access-84bf9\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.966001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.955598    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.966001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.955666    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.966001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.955731    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.991899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.975576    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.991899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.977248    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.991899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.977519    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.991899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.978150    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.991899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.981127    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:29.991899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:29.987380    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:30.003444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:30.003371    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:30.013637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:30.013570    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84bf9\" (UniqueName: \"kubernetes.io/projected/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-kube-api-access-84bf9\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:30.030259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:30.027756    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:30.191272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:30.191010    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:30.285055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:30.284002    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"]
Oct 31 13:28:30.308511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:30.307953    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:30.308511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:30.308048    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:30.308511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:30.308090    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:30.533026 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:30.532999    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"]
Oct 31 13:28:30.533357 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:28:30.533329    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cf573f_d0ac_4fa3_bec6_24715bfdb9b4.slice/crio-edd2a2313808476012738af4f6c2870681528d50611f75b45552f1c951012dda WatchSource:0}: Error finding container edd2a2313808476012738af4f6c2870681528d50611f75b45552f1c951012dda: Status 404 returned error can't find the container with id edd2a2313808476012738af4f6c2870681528d50611f75b45552f1c951012dda
Oct 31 13:28:30.535540 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:30.535499    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:30.535650 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:30.535589    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:30.535650 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:30.535631    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:30.751897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:30.741248    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:31.066900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:31.066835    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6" event={"ID":"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4","Type":"ContainerStarted","Data":"00f748be89a81f9ff06efc0c3f850515ca8fe06d72b33d04e931819ae67f8429"}
Oct 31 13:28:31.067079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:31.066908    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6" event={"ID":"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4","Type":"ContainerStarted","Data":"edd2a2313808476012738af4f6c2870681528d50611f75b45552f1c951012dda"}
Oct 31 13:28:31.069210 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:31.069179    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl" event={"ID":"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6","Type":"ContainerStarted","Data":"48e627edfa7d2dfb092e316b6e688a7f7f5b3bd135cdde5ee01a6d34334adb83"}
Oct 31 13:28:31.069350 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:31.069228    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl" event={"ID":"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6","Type":"ContainerStarted","Data":"9d610183403a6f7e700fb8759af11c6306f8bf8d2e7015e3b407ae263954b6b4"}
Oct 31 13:28:31.095410 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:31.095363    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6" podStartSLOduration=3.095350473 podStartE2EDuration="3.095350473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:31.09523336 +0000 UTC m=+1508.493581983" watchObservedRunningTime="2026-10-31 13:28:31.095350473 +0000 UTC m=+1508.493699095"
Oct 31 13:28:31.127308 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:31.127250    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl" podStartSLOduration=3.127235878 podStartE2EDuration="3.127235878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:31.123243144 +0000 UTC m=+1508.521591769" watchObservedRunningTime="2026-10-31 13:28:31.127235878 +0000 UTC m=+1508.525584547"
Oct 31 13:28:31.191610 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:31.191384    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:31.193437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:31.193410    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 13:28:31.193599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:31.193466    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6" podUID="37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 13:28:31.742559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:31.742521    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:31.808363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:31.806847    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:31.830892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:31.815014    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:28:31.830892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:31.815074    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl" podUID="7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 13:28:32.194510 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:32.194416    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 13:28:32.195124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:32.195098    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6" podUID="37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 13:28:32.804366 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:32.804328    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:28:32.804552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:32.804380    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl" podUID="7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 13:28:33.002892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:33.002321    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:33.195637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:33.195212    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 13:28:33.195637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:33.195260    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6" podUID="37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 13:28:33.804104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:33.804061    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:28:33.804313 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:33.804120    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl" podUID="7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 13:28:34.193302 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:34.193211    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 13:28:34.193565 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:34.193538    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6" podUID="37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 13:28:34.804648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:34.804614    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:28:34.805154 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:34.804667    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl" podUID="7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 13:28:35.194953 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:35.194260    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 13:28:35.194953 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:35.194323    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6" podUID="37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 13:28:35.309733 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:35.309700    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:35.705748 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:35.705716    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:35.811459 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:35.811420    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:28:35.812099 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:35.812059    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl" podUID="7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 13:28:36.193303 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:36.193265    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 13:28:36.193600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:36.193573    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6" podUID="37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 13:28:36.804554 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:36.804517    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:28:36.804748 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:36.804572    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl" podUID="7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 13:28:37.191407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:37.191327    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 13:28:37.191407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:37.191374    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6" podUID="37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 13:28:37.294253 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:37.294219    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:37.804387 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:37.804355    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:28:37.804563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:37.804413    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl" podUID="7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 13:28:38.467015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:38.466974    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:38.468124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:38.468101    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:38.469409 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:38.469379    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:28:38.618886 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:38.618844    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:38.808000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:38.807926    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:38.808394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:38.808375    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:38.809244 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:38.809227    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:28:39.696783 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:39.696745    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:40.060630 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:40.060597    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"]
Oct 31 13:28:40.064938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:40.064920    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"
Oct 31 13:28:40.072965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:40.072703    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:28:40.072965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:40.072800    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:28:40.077534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:40.077510    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"]
Oct 31 13:28:40.202325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:40.202296    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46cd4d85-81a9-44ae-a847-28cb5046e0d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14c80e96-fj8ld\" (UID: \"46cd4d85-81a9-44ae-a847-28cb5046e0d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"
Oct 31 13:28:40.202325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:40.202327    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8chr\" (UniqueName: \"kubernetes.io/projected/46cd4d85-81a9-44ae-a847-28cb5046e0d9-kube-api-access-h8chr\") pod \"maas-api-revoke-sub-14c80e96-fj8ld\" (UID: \"46cd4d85-81a9-44ae-a847-28cb5046e0d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"
Oct 31 13:28:40.202598 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:40.202526    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46cd4d85-81a9-44ae-a847-28cb5046e0d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14c80e96-fj8ld\" (UID: \"46cd4d85-81a9-44ae-a847-28cb5046e0d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"
Oct 31 13:28:40.303887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:40.303836    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46cd4d85-81a9-44ae-a847-28cb5046e0d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14c80e96-fj8ld\" (UID: \"46cd4d85-81a9-44ae-a847-28cb5046e0d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"
Oct 31 13:28:40.304112 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:40.303895    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8chr\" (UniqueName: \"kubernetes.io/projected/46cd4d85-81a9-44ae-a847-28cb5046e0d9-kube-api-access-h8chr\") pod \"maas-api-revoke-sub-14c80e96-fj8ld\" (UID: \"46cd4d85-81a9-44ae-a847-28cb5046e0d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"
Oct 31 13:28:40.304192 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:40.304109    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46cd4d85-81a9-44ae-a847-28cb5046e0d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14c80e96-fj8ld\" (UID: \"46cd4d85-81a9-44ae-a847-28cb5046e0d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"
Oct 31 13:28:40.304681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:40.304650    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46cd4d85-81a9-44ae-a847-28cb5046e0d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14c80e96-fj8ld\" (UID: \"46cd4d85-81a9-44ae-a847-28cb5046e0d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"
Oct 31 13:28:40.306455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:40.306424    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46cd4d85-81a9-44ae-a847-28cb5046e0d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14c80e96-fj8ld\" (UID: \"46cd4d85-81a9-44ae-a847-28cb5046e0d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"
Oct 31 13:28:40.317259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:40.317208    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8chr\" (UniqueName: \"kubernetes.io/projected/46cd4d85-81a9-44ae-a847-28cb5046e0d9-kube-api-access-h8chr\") pod \"maas-api-revoke-sub-14c80e96-fj8ld\" (UID: \"46cd4d85-81a9-44ae-a847-28cb5046e0d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"
Oct 31 13:28:40.381085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:40.381059    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"
Oct 31 13:28:40.665987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:40.665957    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"]
Oct 31 13:28:40.666906 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:28:40.666853    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cd4d85_81a9_44ae_a847_28cb5046e0d9.slice/crio-34006021cf4e9eda7a836301e00f238d6fa5b9ac04487b3df49d7c7fd87a81ff WatchSource:0}: Error finding container 34006021cf4e9eda7a836301e00f238d6fa5b9ac04487b3df49d7c7fd87a81ff: Status 404 returned error can't find the container with id 34006021cf4e9eda7a836301e00f238d6fa5b9ac04487b3df49d7c7fd87a81ff
Oct 31 13:28:40.688293 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:40.688267    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:41.173178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:41.173141    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld" event={"ID":"46cd4d85-81a9-44ae-a847-28cb5046e0d9","Type":"ContainerStarted","Data":"77b85e79d4a3f997795bdf59ebc8b2614da484bad221a20b5ebce63e63558ff3"}
Oct 31 13:28:41.173558 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:41.173208    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld" event={"ID":"46cd4d85-81a9-44ae-a847-28cb5046e0d9","Type":"ContainerStarted","Data":"34006021cf4e9eda7a836301e00f238d6fa5b9ac04487b3df49d7c7fd87a81ff"}
Oct 31 13:28:41.194705 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:41.194653    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld" podStartSLOduration=1.194638578 podStartE2EDuration="1.194638578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:41.192203349 +0000 UTC m=+1518.590551974" watchObservedRunningTime="2026-10-31 13:28:41.194638578 +0000 UTC m=+1518.592987201"
Oct 31 13:28:41.601161 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:41.601122    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:42.180948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:42.180915    4428 generic.go:356] "Generic (PLEG): container finished" podID="46cd4d85-81a9-44ae-a847-28cb5046e0d9" containerID="77b85e79d4a3f997795bdf59ebc8b2614da484bad221a20b5ebce63e63558ff3" exitCode=0
Oct 31 13:28:42.181349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:42.180998    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld" event={"ID":"46cd4d85-81a9-44ae-a847-28cb5046e0d9","Type":"ContainerDied","Data":"77b85e79d4a3f997795bdf59ebc8b2614da484bad221a20b5ebce63e63558ff3"}
Oct 31 13:28:42.523888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:42.523828    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:42.649947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:42.649914    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"]
Oct 31 13:28:42.654028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:42.654007    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"
Oct 31 13:28:42.663404 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:42.663381    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"]
Oct 31 13:28:42.745518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:42.745490    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfpx9\" (UniqueName: \"kubernetes.io/projected/096c7ece-20c0-409f-b7f9-384562f5f452-kube-api-access-rfpx9\") pod \"maas-api-revoke-sub-5a52ccf6-j6v49\" (UID: \"096c7ece-20c0-409f-b7f9-384562f5f452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"
Oct 31 13:28:42.745656 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:42.745537    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096c7ece-20c0-409f-b7f9-384562f5f452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a52ccf6-j6v49\" (UID: \"096c7ece-20c0-409f-b7f9-384562f5f452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"
Oct 31 13:28:42.745656 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:42.745617    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096c7ece-20c0-409f-b7f9-384562f5f452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a52ccf6-j6v49\" (UID: \"096c7ece-20c0-409f-b7f9-384562f5f452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"
Oct 31 13:28:42.846918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:42.846849    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096c7ece-20c0-409f-b7f9-384562f5f452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a52ccf6-j6v49\" (UID: \"096c7ece-20c0-409f-b7f9-384562f5f452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"
Oct 31 13:28:42.847067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:42.846920    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096c7ece-20c0-409f-b7f9-384562f5f452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a52ccf6-j6v49\" (UID: \"096c7ece-20c0-409f-b7f9-384562f5f452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"
Oct 31 13:28:42.847132 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:42.847067    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfpx9\" (UniqueName: \"kubernetes.io/projected/096c7ece-20c0-409f-b7f9-384562f5f452-kube-api-access-rfpx9\") pod \"maas-api-revoke-sub-5a52ccf6-j6v49\" (UID: \"096c7ece-20c0-409f-b7f9-384562f5f452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"
Oct 31 13:28:42.847547 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:42.847521    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096c7ece-20c0-409f-b7f9-384562f5f452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a52ccf6-j6v49\" (UID: \"096c7ece-20c0-409f-b7f9-384562f5f452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"
Oct 31 13:28:42.849357 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:42.849339    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096c7ece-20c0-409f-b7f9-384562f5f452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a52ccf6-j6v49\" (UID: \"096c7ece-20c0-409f-b7f9-384562f5f452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"
Oct 31 13:28:42.859797 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:42.859776    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfpx9\" (UniqueName: \"kubernetes.io/projected/096c7ece-20c0-409f-b7f9-384562f5f452-kube-api-access-rfpx9\") pod \"maas-api-revoke-sub-5a52ccf6-j6v49\" (UID: \"096c7ece-20c0-409f-b7f9-384562f5f452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"
Oct 31 13:28:42.969047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:42.969027    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"
Oct 31 13:28:43.262590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.262558    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"]
Oct 31 13:28:43.334317 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.334291    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"
Oct 31 13:28:43.366840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.366810    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46cd4d85-81a9-44ae-a847-28cb5046e0d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46cd4d85-81a9-44ae-a847-28cb5046e0d9-maas-api-xks-ca\") pod \"46cd4d85-81a9-44ae-a847-28cb5046e0d9\" (UID: \"46cd4d85-81a9-44ae-a847-28cb5046e0d9\") "
Oct 31 13:28:43.366988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.366957    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46cd4d85-81a9-44ae-a847-28cb5046e0d9-kube-api-access-h8chr\" (UniqueName: \"kubernetes.io/projected/46cd4d85-81a9-44ae-a847-28cb5046e0d9-kube-api-access-h8chr\") pod \"46cd4d85-81a9-44ae-a847-28cb5046e0d9\" (UID: \"46cd4d85-81a9-44ae-a847-28cb5046e0d9\") "
Oct 31 13:28:43.367161 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.367144    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46cd4d85-81a9-44ae-a847-28cb5046e0d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46cd4d85-81a9-44ae-a847-28cb5046e0d9-maas-api-openshift-ca\") pod \"46cd4d85-81a9-44ae-a847-28cb5046e0d9\" (UID: \"46cd4d85-81a9-44ae-a847-28cb5046e0d9\") "
Oct 31 13:28:43.367585 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.367558    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46cd4d85-81a9-44ae-a847-28cb5046e0d9-maas-api-openshift-ca" pod "46cd4d85-81a9-44ae-a847-28cb5046e0d9" (UID: "46cd4d85-81a9-44ae-a847-28cb5046e0d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:28:43.369126 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.369100    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46cd4d85-81a9-44ae-a847-28cb5046e0d9-kube-api-access-h8chr" pod "46cd4d85-81a9-44ae-a847-28cb5046e0d9" (UID: "46cd4d85-81a9-44ae-a847-28cb5046e0d9"). InnerVolumeSpecName "kube-api-access-h8chr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:28:43.369797 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.369771    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46cd4d85-81a9-44ae-a847-28cb5046e0d9-maas-api-xks-ca" pod "46cd4d85-81a9-44ae-a847-28cb5046e0d9" (UID: "46cd4d85-81a9-44ae-a847-28cb5046e0d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:28:43.436623 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.424856    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:43.468553 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.468524    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46cd4d85-81a9-44ae-a847-28cb5046e0d9-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:43.490213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.490181    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8chr\" (UniqueName: \"kubernetes.io/projected/46cd4d85-81a9-44ae-a847-28cb5046e0d9-kube-api-access-h8chr\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:43.490436 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.490415    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46cd4d85-81a9-44ae-a847-28cb5046e0d9-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:43.525987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.525959    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"]
Oct 31 13:28:43.526559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.526543    4428 cpu_manager.go:414] "containerMap: removing container" podUID="46cd4d85-81a9-44ae-a847-28cb5046e0d9" containerName="revoke-keys"
Oct 31 13:28:43.526627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.526560    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46cd4d85-81a9-44ae-a847-28cb5046e0d9" containerName="revoke-keys"
Oct 31 13:28:43.526690 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.526656    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="46cd4d85-81a9-44ae-a847-28cb5046e0d9" containerName="revoke-keys"
Oct 31 13:28:43.530252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.530233    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.545704 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.545666    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"]
Oct 31 13:28:43.592117 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.592089    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.592254 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.592119    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.592254 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.592210    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.592350 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.592325    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rslzr\" (UniqueName: \"kubernetes.io/projected/9796198f-74dc-491b-b5fe-7ee7bb9ae809-kube-api-access-rslzr\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.592403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.592387    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.592506 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.592490    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.592558 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.592530    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.592618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.592579    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.592618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.592605    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.668563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.668531    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"]
Oct 31 13:28:43.672699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.672682    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.686463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.685803    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"]
Oct 31 13:28:43.694498 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.694464    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.694625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.694508    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.694625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.694609    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.694744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.694684    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rslzr\" (UniqueName: \"kubernetes.io/projected/9796198f-74dc-491b-b5fe-7ee7bb9ae809-kube-api-access-rslzr\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.694744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.694734    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.694853 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.694832    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.694920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.694886    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.694920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.694911    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.695087 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.694934    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.695724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.695697    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.696086 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.696059    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.696335 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.696312    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.696670 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.696633    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.696761 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.696731    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.701158 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.701139    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.704778 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.704753    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.709326 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.709303    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rslzr\" (UniqueName: \"kubernetes.io/projected/9796198f-74dc-491b-b5fe-7ee7bb9ae809-kube-api-access-rslzr\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.709535 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.709517    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.795995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.795913    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.795995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.795976    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.796220 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.796157    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4853846-205d-4d62-a056-db336dee24cb-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.796220 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.796199    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.796331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.796238    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4853846-205d-4d62-a056-db336dee24cb-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.796380 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.796347    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.796429 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.796403    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4853846-205d-4d62-a056-db336dee24cb-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.796478 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.796442    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.796531 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.796499    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w595z\" (UniqueName: \"kubernetes.io/projected/a4853846-205d-4d62-a056-db336dee24cb-kube-api-access-w595z\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.845372 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.845323    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:43.852305 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.852275    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"]
Oct 31 13:28:43.856467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.856438    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:43.873506 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.872943    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"]
Oct 31 13:28:43.891845 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.891746    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898108    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898162    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898211    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898246    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898287    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898360    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4853846-205d-4d62-a056-db336dee24cb-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898390    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898425    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4853846-205d-4d62-a056-db336dee24cb-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898447    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898500    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898546    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a64a3776-6565-48d0-8c36-5c4d979eeb02-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898573    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898597    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898630    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4853846-205d-4d62-a056-db336dee24cb-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898656    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898670    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898686    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l67qg\" (UniqueName: \"kubernetes.io/projected/a64a3776-6565-48d0-8c36-5c4d979eeb02-kube-api-access-l67qg\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898705    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:43.898840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.898741    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w595z\" (UniqueName: \"kubernetes.io/projected/a4853846-205d-4d62-a056-db336dee24cb-kube-api-access-w595z\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.899850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.899634    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.899850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.899766    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.900002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.899957    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.900167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.900147    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4853846-205d-4d62-a056-db336dee24cb-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.901624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.901599    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4853846-205d-4d62-a056-db336dee24cb-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.911895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.908297    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:43.954046 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:43.952741    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4853846-205d-4d62-a056-db336dee24cb-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:44.037336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.020547    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w595z\" (UniqueName: \"kubernetes.io/projected/a4853846-205d-4d62-a056-db336dee24cb-kube-api-access-w595z\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:44.037336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.021393    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a64a3776-6565-48d0-8c36-5c4d979eeb02-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.037336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.021451    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.037336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.021509    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.037336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.021538    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l67qg\" (UniqueName: \"kubernetes.io/projected/a64a3776-6565-48d0-8c36-5c4d979eeb02-kube-api-access-l67qg\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.037336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.021568    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.037336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.021658    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.037336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.021723    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.037336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.021780    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.037336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.021933    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.062761 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.059472    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.081981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.081788    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:44.083823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.083605    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.084631 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.084603    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a64a3776-6565-48d0-8c36-5c4d979eeb02-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.095454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.088387    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.095454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.088640    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.095454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.090227    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"]
Oct 31 13:28:44.111548 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.106940    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l67qg\" (UniqueName: \"kubernetes.io/projected/a64a3776-6565-48d0-8c36-5c4d979eeb02-kube-api-access-l67qg\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.111548 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.108192    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.111548 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.110200    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.121276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.116599    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"]
Oct 31 13:28:44.121276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.116741    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.133834 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.128996    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.190055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.183069    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"]
Oct 31 13:28:44.190055 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:28:44.183698    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9796198f_74dc_491b_b5fe_7ee7bb9ae809.slice/crio-b4dcdf86f96907844d8e5aaa3ff08034c839ff3c9ff9bce019a813222a4c1cdc WatchSource:0}: Error finding container b4dcdf86f96907844d8e5aaa3ff08034c839ff3c9ff9bce019a813222a4c1cdc: Status 404 returned error can't find the container with id b4dcdf86f96907844d8e5aaa3ff08034c839ff3c9ff9bce019a813222a4c1cdc
Oct 31 13:28:44.190055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.187445    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:44.190055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.187531    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:44.190055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.187580    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:44.198268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.198244    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2" event={"ID":"9796198f-74dc-491b-b5fe-7ee7bb9ae809","Type":"ContainerStarted","Data":"b4dcdf86f96907844d8e5aaa3ff08034c839ff3c9ff9bce019a813222a4c1cdc"}
Oct 31 13:28:44.200336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.200313    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49" event={"ID":"096c7ece-20c0-409f-b7f9-384562f5f452","Type":"ContainerStarted","Data":"a60c43c04dfd83da1843b3a6429d24d45f6f65182e1551e9a8026d708bd1e507"}
Oct 31 13:28:44.201572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.201174    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49" event={"ID":"096c7ece-20c0-409f-b7f9-384562f5f452","Type":"ContainerStarted","Data":"222dac6974dbf13dc7a617aa8eb9b33bd04ef841eac08823aa2576ca68723ff4"}
Oct 31 13:28:44.202024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.202003    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld" event={"ID":"46cd4d85-81a9-44ae-a847-28cb5046e0d9","Type":"ContainerDied","Data":"34006021cf4e9eda7a836301e00f238d6fa5b9ac04487b3df49d7c7fd87a81ff"}
Oct 31 13:28:44.202110 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.202043    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34006021cf4e9eda7a836301e00f238d6fa5b9ac04487b3df49d7c7fd87a81ff"
Oct 31 13:28:44.202359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.202115    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"
Oct 31 13:28:44.244994 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.244940    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49" podStartSLOduration=2.244920376 podStartE2EDuration="2.244920376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:44.228881344 +0000 UTC m=+1521.627229968" watchObservedRunningTime="2026-10-31 13:28:44.244920376 +0000 UTC m=+1521.643268997"
Oct 31 13:28:44.285204 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.285176    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04337deb-7a21-4e58-8a41-098d8be2e918-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.285624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.285307    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.285624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.285352    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-554xx\" (UniqueName: \"kubernetes.io/projected/04337deb-7a21-4e58-8a41-098d8be2e918-kube-api-access-554xx\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.285624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.285451    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.285624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.285571    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04337deb-7a21-4e58-8a41-098d8be2e918-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.285859 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.285626    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.285859 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.285724    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.285859 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.285776    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.286071 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.285952    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04337deb-7a21-4e58-8a41-098d8be2e918-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.368641 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.368534    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:44.386782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.386736    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.386930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.386833    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04337deb-7a21-4e58-8a41-098d8be2e918-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.386930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.386901    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.387068 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.386951    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.387068 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.387024    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.387182 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.387122    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04337deb-7a21-4e58-8a41-098d8be2e918-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.387182 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.387146    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.387182 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.387163    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04337deb-7a21-4e58-8a41-098d8be2e918-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.387354 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.387275    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.387354 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.387306    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-554xx\" (UniqueName: \"kubernetes.io/projected/04337deb-7a21-4e58-8a41-098d8be2e918-kube-api-access-554xx\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.387814 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.387788    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04337deb-7a21-4e58-8a41-098d8be2e918-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.387950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.387910    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.388127 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.388101    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.388313 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.388288    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.390739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.390713    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04337deb-7a21-4e58-8a41-098d8be2e918-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.391127 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.391093    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.402055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.402006    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04337deb-7a21-4e58-8a41-098d8be2e918-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.402601 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.402566    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-554xx\" (UniqueName: \"kubernetes.io/projected/04337deb-7a21-4e58-8a41-098d8be2e918-kube-api-access-554xx\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.407138 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.407120    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"]
Oct 31 13:28:44.409107 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:28:44.409083    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4853846_205d_4d62_a056_db336dee24cb.slice/crio-2830ff2fb5bd14f0bdacd23016342a74a8344a7a00298f6730fd9f5bcdeb676e WatchSource:0}: Error finding container 2830ff2fb5bd14f0bdacd23016342a74a8344a7a00298f6730fd9f5bcdeb676e: Status 404 returned error can't find the container with id 2830ff2fb5bd14f0bdacd23016342a74a8344a7a00298f6730fd9f5bcdeb676e
Oct 31 13:28:44.411756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.411726    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:44.411832 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.411789    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:44.411832 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.411821    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:44.458026 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.457181    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:44.574491 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.574402    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:44.680138 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.680099    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"]
Oct 31 13:28:44.688835 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:28:44.688810    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64a3776_6565_48d0_8c36_5c4d979eeb02.slice/crio-e434e5e14b6269a2c9ff8247e3fc1371599e919ca5d0c18d3509423839056594 WatchSource:0}: Error finding container e434e5e14b6269a2c9ff8247e3fc1371599e919ca5d0c18d3509423839056594: Status 404 returned error can't find the container with id e434e5e14b6269a2c9ff8247e3fc1371599e919ca5d0c18d3509423839056594
Oct 31 13:28:44.692645 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.691506    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:44.692645 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.691591    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:44.692645 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.691635    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:44.889460 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.889427    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"]
Oct 31 13:28:44.891255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.891217    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:44.891368 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.891306    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:44.891368 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:44.891354    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:28:45.217259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.217222    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x" event={"ID":"04337deb-7a21-4e58-8a41-098d8be2e918","Type":"ContainerStarted","Data":"80928b5c5ee0345e1495fb1e3d18877c09c8fa337571687312e97d43a0fc1123"}
Oct 31 13:28:45.217461 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.217289    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x" event={"ID":"04337deb-7a21-4e58-8a41-098d8be2e918","Type":"ContainerStarted","Data":"354a325808963882998d74a64cc90cbc72fda1ad58fc8abe7daae9aedea26c8a"}
Oct 31 13:28:45.219329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.219292    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt" event={"ID":"a64a3776-6565-48d0-8c36-5c4d979eeb02","Type":"ContainerStarted","Data":"20d158b99f4b4906ebae7ac5a24bf7263fade6f80cffe811f49432955e146da3"}
Oct 31 13:28:45.219468 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.219356    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt" event={"ID":"a64a3776-6565-48d0-8c36-5c4d979eeb02","Type":"ContainerStarted","Data":"e434e5e14b6269a2c9ff8247e3fc1371599e919ca5d0c18d3509423839056594"}
Oct 31 13:28:45.221353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.221324    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2" event={"ID":"9796198f-74dc-491b-b5fe-7ee7bb9ae809","Type":"ContainerStarted","Data":"d171285953525cf3a208e44c8fba973804fa5563b3413e5785d39cb6f371afa7"}
Oct 31 13:28:45.224122 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.224097    4428 generic.go:356] "Generic (PLEG): container finished" podID="096c7ece-20c0-409f-b7f9-384562f5f452" containerID="a60c43c04dfd83da1843b3a6429d24d45f6f65182e1551e9a8026d708bd1e507" exitCode=0
Oct 31 13:28:45.224224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.224172    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49" event={"ID":"096c7ece-20c0-409f-b7f9-384562f5f452","Type":"ContainerDied","Data":"a60c43c04dfd83da1843b3a6429d24d45f6f65182e1551e9a8026d708bd1e507"}
Oct 31 13:28:45.227921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.227895    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl" event={"ID":"a4853846-205d-4d62-a056-db336dee24cb","Type":"ContainerStarted","Data":"c87a1754d7427fa96dcfc16168934d30dfde33085f41744efb5f8460046f5e5d"}
Oct 31 13:28:45.228047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.227944    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl" event={"ID":"a4853846-205d-4d62-a056-db336dee24cb","Type":"ContainerStarted","Data":"2830ff2fb5bd14f0bdacd23016342a74a8344a7a00298f6730fd9f5bcdeb676e"}
Oct 31 13:28:45.241108 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.241056    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x" podStartSLOduration=2.241040452 podStartE2EDuration="2.241040452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:45.240349822 +0000 UTC m=+1522.638698446" watchObservedRunningTime="2026-10-31 13:28:45.241040452 +0000 UTC m=+1522.639389077"
Oct 31 13:28:45.288511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.286337    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl" podStartSLOduration=2.286315106 podStartE2EDuration="2.286315106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:45.272905883 +0000 UTC m=+1522.671254505" watchObservedRunningTime="2026-10-31 13:28:45.286315106 +0000 UTC m=+1522.684663733"
Oct 31 13:28:45.341563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.341508    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2" podStartSLOduration=2.341494393 podStartE2EDuration="2.341494393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:45.332569708 +0000 UTC m=+1522.730918333" watchObservedRunningTime="2026-10-31 13:28:45.341494393 +0000 UTC m=+1522.739843017"
Oct 31 13:28:45.376555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.376524    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:45.383287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.383253    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:28:45.383539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.383512    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt" podUID="a64a3776-6565-48d0-8c36-5c4d979eeb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:28:45.436787 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.436727    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt" podStartSLOduration=2.4367061 podStartE2EDuration="2.4367061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:45.376218737 +0000 UTC m=+1522.774567363" watchObservedRunningTime="2026-10-31 13:28:45.4367061 +0000 UTC m=+1522.835054728"
Oct 31 13:28:45.466065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.466037    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:45.514957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.514845    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:45.855946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.854108    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:45.861888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.860821    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:28:45.861888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:45.860900    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2" podUID="9796198f-74dc-491b-b5fe-7ee7bb9ae809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:28:46.106954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:46.106669    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:46.106954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:46.106841    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:28:46.106954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:46.106904    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl" podUID="a4853846-205d-4d62-a056-db336dee24cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:28:46.405895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:46.390316    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:28:46.405895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:46.390421    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt" podUID="a64a3776-6565-48d0-8c36-5c4d979eeb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:28:46.494905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:46.494841    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 13:28:46.520018 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:46.519963    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x" podUID="04337deb-7a21-4e58-8a41-098d8be2e918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 13:28:46.580856 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:46.576606    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:46.949892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:46.944335    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:28:46.949892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:46.944411    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2" podUID="9796198f-74dc-491b-b5fe-7ee7bb9ae809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:28:47.128894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:47.116669    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:28:47.128894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:47.116743    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl" podUID="a4853846-205d-4d62-a056-db336dee24cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:28:47.387899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:47.387241    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:28:47.413186 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:47.411993    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:28:47.666922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:47.643976    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:28:47.740726 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:47.695134    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:47.740726 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:47.710849    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:47.759501 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:47.748600    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:47.802896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:47.783474    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:48.004437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:47.853018    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:48.004437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:47.944034    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:28:48.097771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.097734    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:28:48.164264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.157440    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt" podUID="a64a3776-6565-48d0-8c36-5c4d979eeb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:28:48.182897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.098734    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:28:48.182897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.176476    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2" podUID="9796198f-74dc-491b-b5fe-7ee7bb9ae809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:28:48.226911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.199836    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:28:48.226911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.203010    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:48.228132 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.228058    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl" podUID="a4853846-205d-4d62-a056-db336dee24cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:28:48.261114 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.261019    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 13:28:48.262320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.262280    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x" podUID="04337deb-7a21-4e58-8a41-098d8be2e918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 13:28:48.288921 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:28:48.273213    4428 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.359s"
Oct 31 13:28:48.330894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.330123    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"
Oct 31 13:28:48.347128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.344049    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:48.379616 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.374721    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:28:48.379616 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.374775    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt" podUID="a64a3776-6565-48d0-8c36-5c4d979eeb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:28:48.414790 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.414055    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:48.423894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.415754    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:28:48.423894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.416146    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:28:48.423894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.416201    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:28:48.463895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.455605    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/096c7ece-20c0-409f-b7f9-384562f5f452-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096c7ece-20c0-409f-b7f9-384562f5f452-maas-api-openshift-ca\") pod \"096c7ece-20c0-409f-b7f9-384562f5f452\" (UID: \"096c7ece-20c0-409f-b7f9-384562f5f452\") "
Oct 31 13:28:48.463895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.455808    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096c7ece-20c0-409f-b7f9-384562f5f452-kube-api-access-rfpx9\" (UniqueName: \"kubernetes.io/projected/096c7ece-20c0-409f-b7f9-384562f5f452-kube-api-access-rfpx9\") pod \"096c7ece-20c0-409f-b7f9-384562f5f452\" (UID: \"096c7ece-20c0-409f-b7f9-384562f5f452\") "
Oct 31 13:28:48.464686 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.464661    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/096c7ece-20c0-409f-b7f9-384562f5f452-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096c7ece-20c0-409f-b7f9-384562f5f452-maas-api-xks-ca\") pod \"096c7ece-20c0-409f-b7f9-384562f5f452\" (UID: \"096c7ece-20c0-409f-b7f9-384562f5f452\") "
Oct 31 13:28:48.570364 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.529747    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 13:28:48.570364 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.529802    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x" podUID="04337deb-7a21-4e58-8a41-098d8be2e918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 13:28:48.685801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.649581    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:48.685801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.684407    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:28:48.854947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.711950    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:48.854947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.712386    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:28:48.854947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.828541    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:28:48.854947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.828938    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:28:48.876306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.863212    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:28:48.876306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.863281    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2" podUID="9796198f-74dc-491b-b5fe-7ee7bb9ae809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:28:48.876306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.865812    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Oct 31 13:28:48.876306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.866067    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:48.876306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.867107    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096c7ece-20c0-409f-b7f9-384562f5f452-maas-api-openshift-ca" pod "096c7ece-20c0-409f-b7f9-384562f5f452" (UID: "096c7ece-20c0-409f-b7f9-384562f5f452"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:28:48.924538 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.924429    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:48.925712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.925673    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/096c7ece-20c0-409f-b7f9-384562f5f452-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:48.950054 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.943715    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:48.992895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.977209    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:28:48.992895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:48.977273    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:28:49.038063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.038023    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/096c7ece-20c0-409f-b7f9-384562f5f452-maas-api-xks-ca" pod "096c7ece-20c0-409f-b7f9-384562f5f452" (UID: "096c7ece-20c0-409f-b7f9-384562f5f452"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:28:49.127997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.127909    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096c7ece-20c0-409f-b7f9-384562f5f452-kube-api-access-rfpx9" pod "096c7ece-20c0-409f-b7f9-384562f5f452" (UID: "096c7ece-20c0-409f-b7f9-384562f5f452"). InnerVolumeSpecName "kube-api-access-rfpx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:28:49.128305 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.128278    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:28:49.128495 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.128467    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl" podUID="a4853846-205d-4d62-a056-db336dee24cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:28:49.341700 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.341669    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/096c7ece-20c0-409f-b7f9-384562f5f452-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:49.341950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.341935    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfpx9\" (UniqueName: \"kubernetes.io/projected/096c7ece-20c0-409f-b7f9-384562f5f452-kube-api-access-rfpx9\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:49.391030 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.390559    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:28:49.391324 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.391281    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt" podUID="a64a3776-6565-48d0-8c36-5c4d979eeb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:28:49.453894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.412540    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49" event={"ID":"096c7ece-20c0-409f-b7f9-384562f5f452","Type":"ContainerDied","Data":"222dac6974dbf13dc7a617aa8eb9b33bd04ef841eac08823aa2576ca68723ff4"}
Oct 31 13:28:49.453894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.412599    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="222dac6974dbf13dc7a617aa8eb9b33bd04ef841eac08823aa2576ca68723ff4"
Oct 31 13:28:49.453894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.412674    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"
Oct 31 13:28:49.455275 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.454830    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:28:49.494050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.494014    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:28:49.507469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.507443    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:28:49.566503 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.557196    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 13:28:49.566503 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.557252    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x" podUID="04337deb-7a21-4e58-8a41-098d8be2e918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 13:28:49.663800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.660231    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:28:49.693232 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.668914    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:49.886891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.868027    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:49.886891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.870270    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:28:49.886891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.870322    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2" podUID="9796198f-74dc-491b-b5fe-7ee7bb9ae809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:28:49.924987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.924898    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:28:49.947444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.947404    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:49.948162 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:49.948132    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Oct 31 13:28:50.117074 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:28:50.117031    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:28:50.118653 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.118627    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:28:50.118909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.118885    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl" podUID="a4853846-205d-4d62-a056-db336dee24cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:28:50.336909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.336132    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:50.418896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.418268    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:28:50.418896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.418342    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt" podUID="a64a3776-6565-48d0-8c36-5c4d979eeb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:28:50.467022 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.466974    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:50.468972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.468945    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 13:28:50.469163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.469142    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x" podUID="04337deb-7a21-4e58-8a41-098d8be2e918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 13:28:50.535894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.517821    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:28:50.535894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.522324    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:50.678952 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:28:50.649526    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096c7ece_20c0_409f_b7f9_384562f5f452.slice/crio-222dac6974dbf13dc7a617aa8eb9b33bd04ef841eac08823aa2576ca68723ff4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096c7ece_20c0_409f_b7f9_384562f5f452.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:28:50.681952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.670037    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:28:50.814538 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.813317    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:50.866455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.866421    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:28:50.866771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.866721    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2" podUID="9796198f-74dc-491b-b5fe-7ee7bb9ae809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:28:50.878998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.878946    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:50.929442 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.929319    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:50.948012 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.947951    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:50.990076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:50.990015    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:51.107893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:51.105397    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:28:51.107893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:51.105453    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl" podUID="a4853846-205d-4d62-a056-db336dee24cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:28:51.354345 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:51.354312    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:28:51.371361 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:51.371328    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:28:51.371658 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:51.371620    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt" podUID="a64a3776-6565-48d0-8c36-5c4d979eeb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:28:51.516274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:51.515269    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:28:51.527995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:51.527951    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:51.601972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:51.597525    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 13:28:51.601972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:51.597584    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x" podUID="04337deb-7a21-4e58-8a41-098d8be2e918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 13:28:51.601972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:51.598044    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:28:51.601972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:51.598367    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:28:51.683848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:51.683734    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:51.830145 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:51.830097    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:51.860101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:51.860060    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:51.961005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:51.879320    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:51.986894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:51.933991    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:28:52.039993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.039950    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2" podUID="9796198f-74dc-491b-b5fe-7ee7bb9ae809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:28:52.047600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:51.935010    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:52.078580 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.069495    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:28:52.219539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.219345    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:28:52.219999 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.219970    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl" podUID="a4853846-205d-4d62-a056-db336dee24cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:28:52.297894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.268968    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:28:52.348903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.346459    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:28:52.354156 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.354129    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:28:52.375854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.375723    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:28:52.377202 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.377167    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt" podUID="a64a3776-6565-48d0-8c36-5c4d979eeb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:28:52.494955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.492025    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 13:28:52.494955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.492092    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x" podUID="04337deb-7a21-4e58-8a41-098d8be2e918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 13:28:52.550074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.548743    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:52.584890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.576002    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:28:52.725109 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.688633    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:52.871600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.869923    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:28:52.900279 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.900241    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:28:52.900570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.900539    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2" podUID="9796198f-74dc-491b-b5fe-7ee7bb9ae809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:28:52.917891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:52.910232    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:53.049255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.049218    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:53.108050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.108007    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"]
Oct 31 13:28:53.109489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.109453    4428 cpu_manager.go:414] "containerMap: removing container" podUID="096c7ece-20c0-409f-b7f9-384562f5f452" containerName="revoke-keys"
Oct 31 13:28:53.110922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.110900    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096c7ece-20c0-409f-b7f9-384562f5f452" containerName="revoke-keys"
Oct 31 13:28:53.111240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.111222    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="096c7ece-20c0-409f-b7f9-384562f5f452" containerName="revoke-keys"
Oct 31 13:28:53.114983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.114957    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:28:53.115152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.115125    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl" podUID="a4853846-205d-4d62-a056-db336dee24cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:28:53.125508 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.125448    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:28:53.127000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.126982    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"]
Oct 31 13:28:53.127920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.126181    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"
Oct 31 13:28:53.141006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.140130    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:28:53.141006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.140953    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:28:53.153992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.153673    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:28:53.251707 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.251650    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84zj8\" (UniqueName: \"kubernetes.io/projected/6aa77e84-829b-49e8-8f79-8aebf5894b42-kube-api-access-84zj8\") pod \"maas-api-revoke-sub-2302af36-fbbpr\" (UID: \"6aa77e84-829b-49e8-8f79-8aebf5894b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"
Oct 31 13:28:53.252108 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.252068    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa77e84-829b-49e8-8f79-8aebf5894b42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2302af36-fbbpr\" (UID: \"6aa77e84-829b-49e8-8f79-8aebf5894b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"
Oct 31 13:28:53.252466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.252443    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa77e84-829b-49e8-8f79-8aebf5894b42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2302af36-fbbpr\" (UID: \"6aa77e84-829b-49e8-8f79-8aebf5894b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"
Oct 31 13:28:53.305900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.302533    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:28:53.385947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.360900    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa77e84-829b-49e8-8f79-8aebf5894b42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2302af36-fbbpr\" (UID: \"6aa77e84-829b-49e8-8f79-8aebf5894b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"
Oct 31 13:28:53.423161 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.423118    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa77e84-829b-49e8-8f79-8aebf5894b42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2302af36-fbbpr\" (UID: \"6aa77e84-829b-49e8-8f79-8aebf5894b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"
Oct 31 13:28:53.423578 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.423552    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84zj8\" (UniqueName: \"kubernetes.io/projected/6aa77e84-829b-49e8-8f79-8aebf5894b42-kube-api-access-84zj8\") pod \"maas-api-revoke-sub-2302af36-fbbpr\" (UID: \"6aa77e84-829b-49e8-8f79-8aebf5894b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"
Oct 31 13:28:53.424353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.384369    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa77e84-829b-49e8-8f79-8aebf5894b42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2302af36-fbbpr\" (UID: \"6aa77e84-829b-49e8-8f79-8aebf5894b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"
Oct 31 13:28:53.430856 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.430832    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa77e84-829b-49e8-8f79-8aebf5894b42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2302af36-fbbpr\" (UID: \"6aa77e84-829b-49e8-8f79-8aebf5894b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"
Oct 31 13:28:53.431039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.384680    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:28:53.432221 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.392487    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:28:53.432415 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.432390    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt" podUID="a64a3776-6565-48d0-8c36-5c4d979eeb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:28:53.512641 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.511340    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 13:28:53.512641 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.511402    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x" podUID="04337deb-7a21-4e58-8a41-098d8be2e918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 13:28:53.520151 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.520105    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84zj8\" (UniqueName: \"kubernetes.io/projected/6aa77e84-829b-49e8-8f79-8aebf5894b42-kube-api-access-84zj8\") pod \"maas-api-revoke-sub-2302af36-fbbpr\" (UID: \"6aa77e84-829b-49e8-8f79-8aebf5894b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"
Oct 31 13:28:53.692534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.692419    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:53.774171 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.774131    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"
Oct 31 13:28:53.853145 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.852249    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:53.853329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.853252    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:53.854344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:53.854283    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:28:54.055649 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.055614    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"]
Oct 31 13:28:54.063071 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.063041    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"
Oct 31 13:28:54.080894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.069397    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"]
Oct 31 13:28:54.142729 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.142701    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"]
Oct 31 13:28:54.144996 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:28:54.144969    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa77e84_829b_49e8_8f79_8aebf5894b42.slice/crio-2456e61718f980a528efc4ca12a3a06c218ed9b0032585fd0202bee45e3ae679 WatchSource:0}: Error finding container 2456e61718f980a528efc4ca12a3a06c218ed9b0032585fd0202bee45e3ae679: Status 404 returned error can't find the container with id 2456e61718f980a528efc4ca12a3a06c218ed9b0032585fd0202bee45e3ae679
Oct 31 13:28:54.163144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.159060    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmwrw\" (UniqueName: \"kubernetes.io/projected/95b31f49-3da5-4a8f-a226-42b79105c854-kube-api-access-hmwrw\") pod \"maas-api-revoke-sub-76da4665-qbwf5\" (UID: \"95b31f49-3da5-4a8f-a226-42b79105c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"
Oct 31 13:28:54.163144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.159143    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b31f49-3da5-4a8f-a226-42b79105c854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76da4665-qbwf5\" (UID: \"95b31f49-3da5-4a8f-a226-42b79105c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"
Oct 31 13:28:54.163144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.159322    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b31f49-3da5-4a8f-a226-42b79105c854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76da4665-qbwf5\" (UID: \"95b31f49-3da5-4a8f-a226-42b79105c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"
Oct 31 13:28:54.199794 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.199765    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:54.200158 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.200133    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:54.201122 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.201102    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:28:54.262451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.262400    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmwrw\" (UniqueName: \"kubernetes.io/projected/95b31f49-3da5-4a8f-a226-42b79105c854-kube-api-access-hmwrw\") pod \"maas-api-revoke-sub-76da4665-qbwf5\" (UID: \"95b31f49-3da5-4a8f-a226-42b79105c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"
Oct 31 13:28:54.262639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.262478    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b31f49-3da5-4a8f-a226-42b79105c854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76da4665-qbwf5\" (UID: \"95b31f49-3da5-4a8f-a226-42b79105c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"
Oct 31 13:28:54.262639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.262583    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b31f49-3da5-4a8f-a226-42b79105c854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76da4665-qbwf5\" (UID: \"95b31f49-3da5-4a8f-a226-42b79105c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"
Oct 31 13:28:54.263582 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.263555    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b31f49-3da5-4a8f-a226-42b79105c854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76da4665-qbwf5\" (UID: \"95b31f49-3da5-4a8f-a226-42b79105c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"
Oct 31 13:28:54.345924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.345832    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b31f49-3da5-4a8f-a226-42b79105c854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76da4665-qbwf5\" (UID: \"95b31f49-3da5-4a8f-a226-42b79105c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"
Oct 31 13:28:54.346116 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.346098    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmwrw\" (UniqueName: \"kubernetes.io/projected/95b31f49-3da5-4a8f-a226-42b79105c854-kube-api-access-hmwrw\") pod \"maas-api-revoke-sub-76da4665-qbwf5\" (UID: \"95b31f49-3da5-4a8f-a226-42b79105c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"
Oct 31 13:28:54.393441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.393387    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"
Oct 31 13:28:54.464810 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.464557    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 13:28:54.464810 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.464617    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x" podUID="04337deb-7a21-4e58-8a41-098d8be2e918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 13:28:54.601111 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.600562    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:54.717982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.717378    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"]
Oct 31 13:28:54.718948 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:28:54.718894    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b31f49_3da5_4a8f_a226_42b79105c854.slice/crio-0b1053d20588503dced8e2938ee628cc581455eb3588780fd6eef0fd4e795ebd WatchSource:0}: Error finding container 0b1053d20588503dced8e2938ee628cc581455eb3588780fd6eef0fd4e795ebd: Status 404 returned error can't find the container with id 0b1053d20588503dced8e2938ee628cc581455eb3588780fd6eef0fd4e795ebd
Oct 31 13:28:54.977566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.977525    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5" event={"ID":"95b31f49-3da5-4a8f-a226-42b79105c854","Type":"ContainerStarted","Data":"0b1053d20588503dced8e2938ee628cc581455eb3588780fd6eef0fd4e795ebd"}
Oct 31 13:28:54.980960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.979291    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr" event={"ID":"6aa77e84-829b-49e8-8f79-8aebf5894b42","Type":"ContainerStarted","Data":"3c1d7a16430b3e57e822b84e77ca074406aec9ce0308124b76650db09c795e32"}
Oct 31 13:28:54.981108 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.980999    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr" event={"ID":"6aa77e84-829b-49e8-8f79-8aebf5894b42","Type":"ContainerStarted","Data":"2456e61718f980a528efc4ca12a3a06c218ed9b0032585fd0202bee45e3ae679"}
Oct 31 13:28:54.986911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.986852    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:54.987198 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.987181    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:54.988190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:54.988171    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:28:55.002271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:55.002223    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr" podStartSLOduration=3.002210276 podStartE2EDuration="3.002210276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:54.99955404 +0000 UTC m=+1532.397902664" watchObservedRunningTime="2026-10-31 13:28:55.002210276 +0000 UTC m=+1532.400558922"
Oct 31 13:28:55.421906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:55.421789    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:55.494734 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:55.494708    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:55.495566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:55.495537    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:55.497218 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:55.497195    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:28:55.988017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:55.987982    4428 generic.go:356] "Generic (PLEG): container finished" podID="6aa77e84-829b-49e8-8f79-8aebf5894b42" containerID="3c1d7a16430b3e57e822b84e77ca074406aec9ce0308124b76650db09c795e32" exitCode=0
Oct 31 13:28:55.988458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:55.988077    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr" event={"ID":"6aa77e84-829b-49e8-8f79-8aebf5894b42","Type":"ContainerDied","Data":"3c1d7a16430b3e57e822b84e77ca074406aec9ce0308124b76650db09c795e32"}
Oct 31 13:28:55.989781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:55.989759    4428 generic.go:356] "Generic (PLEG): container finished" podID="95b31f49-3da5-4a8f-a226-42b79105c854" containerID="e643d2c2f076ad3ace7e1b7b780dc17597394c91bfc0ce6b9889f3809d48eb67" exitCode=0
Oct 31 13:28:55.989935 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:55.989845    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5" event={"ID":"95b31f49-3da5-4a8f-a226-42b79105c854","Type":"ContainerDied","Data":"e643d2c2f076ad3ace7e1b7b780dc17597394c91bfc0ce6b9889f3809d48eb67"}
Oct 31 13:28:56.330891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:56.325997    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:57.213352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.213329    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"
Oct 31 13:28:57.216466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.216447    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"
Oct 31 13:28:57.527511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.522091    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b31f49-3da5-4a8f-a226-42b79105c854-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b31f49-3da5-4a8f-a226-42b79105c854-maas-api-openshift-ca\") pod \"95b31f49-3da5-4a8f-a226-42b79105c854\" (UID: \"95b31f49-3da5-4a8f-a226-42b79105c854\") "
Oct 31 13:28:57.527511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.522217    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b31f49-3da5-4a8f-a226-42b79105c854-kube-api-access-hmwrw\" (UniqueName: \"kubernetes.io/projected/95b31f49-3da5-4a8f-a226-42b79105c854-kube-api-access-hmwrw\") pod \"95b31f49-3da5-4a8f-a226-42b79105c854\" (UID: \"95b31f49-3da5-4a8f-a226-42b79105c854\") "
Oct 31 13:28:57.527511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.522301    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95b31f49-3da5-4a8f-a226-42b79105c854-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b31f49-3da5-4a8f-a226-42b79105c854-maas-api-xks-ca\") pod \"95b31f49-3da5-4a8f-a226-42b79105c854\" (UID: \"95b31f49-3da5-4a8f-a226-42b79105c854\") "
Oct 31 13:28:57.527511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.522375    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa77e84-829b-49e8-8f79-8aebf5894b42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa77e84-829b-49e8-8f79-8aebf5894b42-maas-api-openshift-ca\") pod \"6aa77e84-829b-49e8-8f79-8aebf5894b42\" (UID: \"6aa77e84-829b-49e8-8f79-8aebf5894b42\") "
Oct 31 13:28:57.527511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.522407    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aa77e84-829b-49e8-8f79-8aebf5894b42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa77e84-829b-49e8-8f79-8aebf5894b42-maas-api-xks-ca\") pod \"6aa77e84-829b-49e8-8f79-8aebf5894b42\" (UID: \"6aa77e84-829b-49e8-8f79-8aebf5894b42\") "
Oct 31 13:28:57.527511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.522558    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa77e84-829b-49e8-8f79-8aebf5894b42-kube-api-access-84zj8\" (UniqueName: \"kubernetes.io/projected/6aa77e84-829b-49e8-8f79-8aebf5894b42-kube-api-access-84zj8\") pod \"6aa77e84-829b-49e8-8f79-8aebf5894b42\" (UID: \"6aa77e84-829b-49e8-8f79-8aebf5894b42\") "
Oct 31 13:28:57.534899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.534336    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b31f49-3da5-4a8f-a226-42b79105c854-kube-api-access-hmwrw" pod "95b31f49-3da5-4a8f-a226-42b79105c854" (UID: "95b31f49-3da5-4a8f-a226-42b79105c854"). InnerVolumeSpecName "kube-api-access-hmwrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:28:57.535337 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.535162    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b31f49-3da5-4a8f-a226-42b79105c854-maas-api-openshift-ca" pod "95b31f49-3da5-4a8f-a226-42b79105c854" (UID: "95b31f49-3da5-4a8f-a226-42b79105c854"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:28:57.535992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.535970    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa77e84-829b-49e8-8f79-8aebf5894b42-maas-api-openshift-ca" pod "6aa77e84-829b-49e8-8f79-8aebf5894b42" (UID: "6aa77e84-829b-49e8-8f79-8aebf5894b42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:28:57.547056 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.544961    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa77e84-829b-49e8-8f79-8aebf5894b42-maas-api-xks-ca" pod "6aa77e84-829b-49e8-8f79-8aebf5894b42" (UID: "6aa77e84-829b-49e8-8f79-8aebf5894b42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:28:57.557992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.547691    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:28:57.557992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.554023    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa77e84-829b-49e8-8f79-8aebf5894b42-kube-api-access-84zj8" pod "6aa77e84-829b-49e8-8f79-8aebf5894b42" (UID: "6aa77e84-829b-49e8-8f79-8aebf5894b42"). InnerVolumeSpecName "kube-api-access-84zj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:28:57.564779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.564718    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95b31f49-3da5-4a8f-a226-42b79105c854-maas-api-xks-ca" pod "95b31f49-3da5-4a8f-a226-42b79105c854" (UID: "95b31f49-3da5-4a8f-a226-42b79105c854"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:28:57.624016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.623967    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa77e84-829b-49e8-8f79-8aebf5894b42-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:57.624016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.624007    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa77e84-829b-49e8-8f79-8aebf5894b42-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:57.624016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.624018    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84zj8\" (UniqueName: \"kubernetes.io/projected/6aa77e84-829b-49e8-8f79-8aebf5894b42-kube-api-access-84zj8\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:57.624016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.624027    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b31f49-3da5-4a8f-a226-42b79105c854-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:57.624016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.624036    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmwrw\" (UniqueName: \"kubernetes.io/projected/95b31f49-3da5-4a8f-a226-42b79105c854-kube-api-access-hmwrw\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:57.624412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:57.624053    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b31f49-3da5-4a8f-a226-42b79105c854-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:58.214412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:58.214371    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5" event={"ID":"95b31f49-3da5-4a8f-a226-42b79105c854","Type":"ContainerDied","Data":"0b1053d20588503dced8e2938ee628cc581455eb3588780fd6eef0fd4e795ebd"}
Oct 31 13:28:58.215195 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:58.215173    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b1053d20588503dced8e2938ee628cc581455eb3588780fd6eef0fd4e795ebd"
Oct 31 13:28:58.215432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:58.215417    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"
Oct 31 13:28:58.233169 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:58.233133    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr" event={"ID":"6aa77e84-829b-49e8-8f79-8aebf5894b42","Type":"ContainerDied","Data":"2456e61718f980a528efc4ca12a3a06c218ed9b0032585fd0202bee45e3ae679"}
Oct 31 13:28:58.237924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:58.233848    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2456e61718f980a528efc4ca12a3a06c218ed9b0032585fd0202bee45e3ae679"
Oct 31 13:28:58.238128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:58.233763    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"
Oct 31 13:28:58.797181 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:58.796993    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:58.951038 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:58.950991    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:58.976904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:58.975032    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:59.101140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:59.101035    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:59.298045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:59.297996    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:59.411106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:59.410995    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:28:59.458062 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:59.458003    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:59.670956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:59.666982    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:59.878910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:59.873966    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:59.948958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:59.945975    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:59.967277 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:59.967195    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:59.970901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:59.967961    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:28:59.975997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:59.975964    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:59.976777 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:28:59.976756    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:29:00.120902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:00.101958    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:00.120902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:00.102718    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:29:00.413951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:00.347056    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:00.413951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:00.347540    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:29:00.438912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:00.420969    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:29:00.593152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:00.490001    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:29:00.593470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:00.593444    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:29:00.593652 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:00.593637    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:29:00.668270 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:00.668172    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:00.690400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:00.690368    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:29:00.881895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:00.878265    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:00.966637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:00.966602    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:29:00.967854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:00.967824    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:00.983887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:00.983824    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:01.005023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:01.004971    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:01.111196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:01.111142    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:01.495954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:01.495912    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:29:01.603896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:01.595941    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:01.606031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:01.604245    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:02.055348 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:01.979389    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:02.055348 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:02.040176    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:02.055348 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:02.041243    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:02.065407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:02.065356    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:02.414957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:02.414727    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:29:02.513917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:02.462345    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:02.516586 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:02.516543    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:02.537101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:02.535484    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:29:02.751973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:02.746651    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:29:02.751973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:02.747117    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6" podUID="149a9b52-ef95-4569-b947-e6e14d707339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Oct 31 13:29:02.901936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:02.897986    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:03.047833 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:03.041588    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:29:03.077909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:03.065371    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:03.077909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:03.076591    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:29:03.439422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:03.439307    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:03.474587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:03.474533    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:03.475298 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:03.472340    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:29:03.534144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:03.534000    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:03.761133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:03.761084    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:03.909720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:03.909678    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:03.929724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:03.929686    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:04.047383 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.047280    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:04.066097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.066033    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:04.083908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.081175    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:29:04.232070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.232038    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:29:04.242457 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.241654    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:29:04.317679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.314794    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:29:04.343431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.343404    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:29:04.385302 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.385268    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"]
Oct 31 13:29:04.412895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.386413    4428 cpu_manager.go:414] "containerMap: removing container" podUID="6aa77e84-829b-49e8-8f79-8aebf5894b42" containerName="revoke-keys"
Oct 31 13:29:04.452292 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.444957    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa77e84-829b-49e8-8f79-8aebf5894b42" containerName="revoke-keys"
Oct 31 13:29:04.452531 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.452512    4428 cpu_manager.go:414] "containerMap: removing container" podUID="95b31f49-3da5-4a8f-a226-42b79105c854" containerName="revoke-keys"
Oct 31 13:29:04.452617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.452606    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b31f49-3da5-4a8f-a226-42b79105c854" containerName="revoke-keys"
Oct 31 13:29:04.464928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.452954    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa77e84-829b-49e8-8f79-8aebf5894b42" containerName="revoke-keys"
Oct 31 13:29:04.465125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.465107    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b31f49-3da5-4a8f-a226-42b79105c854" containerName="revoke-keys"
Oct 31 13:29:04.506974 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.504428    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:29:04.511101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.511079    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:29:04.519010 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.518986    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"]
Oct 31 13:29:04.521495 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.506237    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"
Oct 31 13:29:04.526161 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.526143    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:29:04.532149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.529136    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:29:04.532149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.529230    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:29:04.646998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.646917    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852864de-714b-4f59-98b9-e92da30271b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da18ed15-djtsk\" (UID: \"852864de-714b-4f59-98b9-e92da30271b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"
Oct 31 13:29:04.647353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.647064    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ztsk\" (UniqueName: \"kubernetes.io/projected/852864de-714b-4f59-98b9-e92da30271b1-kube-api-access-9ztsk\") pod \"maas-api-revoke-sub-da18ed15-djtsk\" (UID: \"852864de-714b-4f59-98b9-e92da30271b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"
Oct 31 13:29:04.647430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.647412    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852864de-714b-4f59-98b9-e92da30271b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da18ed15-djtsk\" (UID: \"852864de-714b-4f59-98b9-e92da30271b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"
Oct 31 13:29:04.760901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.759903    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852864de-714b-4f59-98b9-e92da30271b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da18ed15-djtsk\" (UID: \"852864de-714b-4f59-98b9-e92da30271b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"
Oct 31 13:29:04.760901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.759996    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ztsk\" (UniqueName: \"kubernetes.io/projected/852864de-714b-4f59-98b9-e92da30271b1-kube-api-access-9ztsk\") pod \"maas-api-revoke-sub-da18ed15-djtsk\" (UID: \"852864de-714b-4f59-98b9-e92da30271b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"
Oct 31 13:29:04.760901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.760153    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852864de-714b-4f59-98b9-e92da30271b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da18ed15-djtsk\" (UID: \"852864de-714b-4f59-98b9-e92da30271b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"
Oct 31 13:29:04.763998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.763970    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852864de-714b-4f59-98b9-e92da30271b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da18ed15-djtsk\" (UID: \"852864de-714b-4f59-98b9-e92da30271b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"
Oct 31 13:29:04.766471 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.766447    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852864de-714b-4f59-98b9-e92da30271b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da18ed15-djtsk\" (UID: \"852864de-714b-4f59-98b9-e92da30271b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"
Oct 31 13:29:04.768176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.768138    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:04.806809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.806763    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ztsk\" (UniqueName: \"kubernetes.io/projected/852864de-714b-4f59-98b9-e92da30271b1-kube-api-access-9ztsk\") pod \"maas-api-revoke-sub-da18ed15-djtsk\" (UID: \"852864de-714b-4f59-98b9-e92da30271b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"
Oct 31 13:29:04.866641 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:04.866610    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"
Oct 31 13:29:05.151217 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:05.151190    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"]
Oct 31 13:29:05.151531 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:29:05.151506    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852864de_714b_4f59_98b9_e92da30271b1.slice/crio-b02a2bc9295ced6279444f2ea215aa230e51ce07861f35f2eee199a73933e695 WatchSource:0}: Error finding container b02a2bc9295ced6279444f2ea215aa230e51ce07861f35f2eee199a73933e695: Status 404 returned error can't find the container with id b02a2bc9295ced6279444f2ea215aa230e51ce07861f35f2eee199a73933e695
Oct 31 13:29:05.790156 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:05.790122    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk" event={"ID":"852864de-714b-4f59-98b9-e92da30271b1","Type":"ContainerStarted","Data":"a3bdec267cbb1e734c67f68c00a2d1fe31caf1f6e1ce83f2dd885359070fdb69"}
Oct 31 13:29:05.790514 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:05.790183    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk" event={"ID":"852864de-714b-4f59-98b9-e92da30271b1","Type":"ContainerStarted","Data":"b02a2bc9295ced6279444f2ea215aa230e51ce07861f35f2eee199a73933e695"}
Oct 31 13:29:05.812775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:05.812728    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk" podStartSLOduration=1.8127140370000001 podStartE2EDuration="1.812714037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:05.808223401 +0000 UTC m=+1543.206572027" watchObservedRunningTime="2026-10-31 13:29:05.812714037 +0000 UTC m=+1543.211062711"
Oct 31 13:29:05.832060 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:05.832032    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:06.436955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:06.420893    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:06.798660 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:06.798630    4428 generic.go:356] "Generic (PLEG): container finished" podID="852864de-714b-4f59-98b9-e92da30271b1" containerID="a3bdec267cbb1e734c67f68c00a2d1fe31caf1f6e1ce83f2dd885359070fdb69" exitCode=0
Oct 31 13:29:06.799050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:06.798667    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk" event={"ID":"852864de-714b-4f59-98b9-e92da30271b1","Type":"ContainerDied","Data":"a3bdec267cbb1e734c67f68c00a2d1fe31caf1f6e1ce83f2dd885359070fdb69"}
Oct 31 13:29:07.215883 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:07.215827    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:07.954175 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:07.954153    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"
Oct 31 13:29:08.003356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:08.003321    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/852864de-714b-4f59-98b9-e92da30271b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852864de-714b-4f59-98b9-e92da30271b1-maas-api-xks-ca\") pod \"852864de-714b-4f59-98b9-e92da30271b1\" (UID: \"852864de-714b-4f59-98b9-e92da30271b1\") "
Oct 31 13:29:08.003524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:08.003421    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852864de-714b-4f59-98b9-e92da30271b1-kube-api-access-9ztsk\" (UniqueName: \"kubernetes.io/projected/852864de-714b-4f59-98b9-e92da30271b1-kube-api-access-9ztsk\") pod \"852864de-714b-4f59-98b9-e92da30271b1\" (UID: \"852864de-714b-4f59-98b9-e92da30271b1\") "
Oct 31 13:29:08.003698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:08.003679    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852864de-714b-4f59-98b9-e92da30271b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852864de-714b-4f59-98b9-e92da30271b1-maas-api-openshift-ca\") pod \"852864de-714b-4f59-98b9-e92da30271b1\" (UID: \"852864de-714b-4f59-98b9-e92da30271b1\") "
Oct 31 13:29:08.004070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:08.004040    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852864de-714b-4f59-98b9-e92da30271b1-maas-api-openshift-ca" pod "852864de-714b-4f59-98b9-e92da30271b1" (UID: "852864de-714b-4f59-98b9-e92da30271b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:29:08.004386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:08.004365    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852864de-714b-4f59-98b9-e92da30271b1-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:08.005915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:08.005860    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/852864de-714b-4f59-98b9-e92da30271b1-maas-api-xks-ca" pod "852864de-714b-4f59-98b9-e92da30271b1" (UID: "852864de-714b-4f59-98b9-e92da30271b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:29:08.006226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:08.006203    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852864de-714b-4f59-98b9-e92da30271b1-kube-api-access-9ztsk" pod "852864de-714b-4f59-98b9-e92da30271b1" (UID: "852864de-714b-4f59-98b9-e92da30271b1"). InnerVolumeSpecName "kube-api-access-9ztsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:29:08.104026 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:08.103833    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:08.113379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:08.112936    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852864de-714b-4f59-98b9-e92da30271b1-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:08.113379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:08.112974    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ztsk\" (UniqueName: \"kubernetes.io/projected/852864de-714b-4f59-98b9-e92da30271b1-kube-api-access-9ztsk\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:08.814073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:08.814040    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"
Oct 31 13:29:08.814073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:08.814068    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk" event={"ID":"852864de-714b-4f59-98b9-e92da30271b1","Type":"ContainerDied","Data":"b02a2bc9295ced6279444f2ea215aa230e51ce07861f35f2eee199a73933e695"}
Oct 31 13:29:08.814288 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:08.814114    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b02a2bc9295ced6279444f2ea215aa230e51ce07861f35f2eee199a73933e695"
Oct 31 13:29:08.923892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:08.921464    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:09.833395 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:09.833348    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:10.795974 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:10.795931    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:11.701433 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:11.701394    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:12.424646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:12.424614    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:12.666612 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:12.666572    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:12.801650 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:12.801619    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"]
Oct 31 13:29:12.802219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:12.802204    4428 cpu_manager.go:414] "containerMap: removing container" podUID="852864de-714b-4f59-98b9-e92da30271b1" containerName="revoke-keys"
Oct 31 13:29:12.802260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:12.802221    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852864de-714b-4f59-98b9-e92da30271b1" containerName="revoke-keys"
Oct 31 13:29:12.802365 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:12.802356    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="852864de-714b-4f59-98b9-e92da30271b1" containerName="revoke-keys"
Oct 31 13:29:12.805677 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:12.805660    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"
Oct 31 13:29:12.810981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:12.810949    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:29:12.811182 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:12.811035    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:29:12.815686 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:12.815664    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"]
Oct 31 13:29:12.899757 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:12.899728    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae32076-375d-48d2-8640-4e9ee3913caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a616a32-xmx68\" (UID: \"6ae32076-375d-48d2-8640-4e9ee3913caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"
Oct 31 13:29:12.899955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:12.899932    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae32076-375d-48d2-8640-4e9ee3913caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a616a32-xmx68\" (UID: \"6ae32076-375d-48d2-8640-4e9ee3913caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"
Oct 31 13:29:12.900101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:12.900081    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th2hc\" (UniqueName: \"kubernetes.io/projected/6ae32076-375d-48d2-8640-4e9ee3913caf-kube-api-access-th2hc\") pod \"maas-api-revoke-sub-0a616a32-xmx68\" (UID: \"6ae32076-375d-48d2-8640-4e9ee3913caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"
Oct 31 13:29:13.001746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.001712    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae32076-375d-48d2-8640-4e9ee3913caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a616a32-xmx68\" (UID: \"6ae32076-375d-48d2-8640-4e9ee3913caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"
Oct 31 13:29:13.001966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.001798    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th2hc\" (UniqueName: \"kubernetes.io/projected/6ae32076-375d-48d2-8640-4e9ee3913caf-kube-api-access-th2hc\") pod \"maas-api-revoke-sub-0a616a32-xmx68\" (UID: \"6ae32076-375d-48d2-8640-4e9ee3913caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"
Oct 31 13:29:13.001966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.001882    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae32076-375d-48d2-8640-4e9ee3913caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a616a32-xmx68\" (UID: \"6ae32076-375d-48d2-8640-4e9ee3913caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"
Oct 31 13:29:13.002540 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.002520    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae32076-375d-48d2-8640-4e9ee3913caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a616a32-xmx68\" (UID: \"6ae32076-375d-48d2-8640-4e9ee3913caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"
Oct 31 13:29:13.004409 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.004384    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae32076-375d-48d2-8640-4e9ee3913caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a616a32-xmx68\" (UID: \"6ae32076-375d-48d2-8640-4e9ee3913caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"
Oct 31 13:29:13.014718 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.014692    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th2hc\" (UniqueName: \"kubernetes.io/projected/6ae32076-375d-48d2-8640-4e9ee3913caf-kube-api-access-th2hc\") pod \"maas-api-revoke-sub-0a616a32-xmx68\" (UID: \"6ae32076-375d-48d2-8640-4e9ee3913caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"
Oct 31 13:29:13.121068 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.120995    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"
Oct 31 13:29:13.413469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.413164    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"]
Oct 31 13:29:13.578699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.578673    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"]
Oct 31 13:29:13.584104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.584085    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.598379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.598352    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"]
Oct 31 13:29:13.631932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.631907    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"]
Oct 31 13:29:13.637618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.637596    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.647091 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.646790    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"]
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.716275    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69e04722-598c-4608-9742-440e327e389a-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.716344    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69e04722-598c-4608-9742-440e327e389a-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.716400    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.716440    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.716494    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.716526    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.716563    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6bn6\" (UniqueName: \"kubernetes.io/projected/69e04722-598c-4608-9742-440e327e389a-kube-api-access-q6bn6\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.716609    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.716670    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.716731    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.716772    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sst8q\" (UniqueName: \"kubernetes.io/projected/af5e815e-eb38-40c4-9651-e4dbbb92dc36-kube-api-access-sst8q\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.716810    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.716841    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.716906    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.716949    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.717008    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.717076    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.725945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.717100    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69e04722-598c-4608-9742-440e327e389a-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.747360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.737677    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:13.817977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.817939    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.818434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818002    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.818434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818023    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69e04722-598c-4608-9742-440e327e389a-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.818434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818123    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69e04722-598c-4608-9742-440e327e389a-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.818434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818166    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69e04722-598c-4608-9742-440e327e389a-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.818434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818229    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.818434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818272    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.818434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818337    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.818434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818345    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.818434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818368    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.818434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818409    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6bn6\" (UniqueName: \"kubernetes.io/projected/69e04722-598c-4608-9742-440e327e389a-kube-api-access-q6bn6\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.819043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818461    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.819043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818477    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.819043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818530    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.819043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818562    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.819043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818600    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sst8q\" (UniqueName: \"kubernetes.io/projected/af5e815e-eb38-40c4-9651-e4dbbb92dc36-kube-api-access-sst8q\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.819043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818640    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.819043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818674    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.819043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818721    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.819043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818757    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.819043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818766    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.819043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.818936    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69e04722-598c-4608-9742-440e327e389a-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.819595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.819059    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.819595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.819095    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.819595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.819264    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.819595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.819373    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.820003 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.819932    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.820121 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.820071    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.821550 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.821523    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.821652 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.821570    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.821713 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.821669    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69e04722-598c-4608-9742-440e327e389a-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.822010 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.821992    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.833779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.833754    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.833990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.833971    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sst8q\" (UniqueName: \"kubernetes.io/projected/af5e815e-eb38-40c4-9651-e4dbbb92dc36-kube-api-access-sst8q\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:13.834071 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.834023    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69e04722-598c-4608-9742-440e327e389a-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.834164 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.834148    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6bn6\" (UniqueName: \"kubernetes.io/projected/69e04722-598c-4608-9742-440e327e389a-kube-api-access-q6bn6\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.899851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.899818    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:13.903147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.903041    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68" event={"ID":"6ae32076-375d-48d2-8640-4e9ee3913caf","Type":"ContainerStarted","Data":"353a7e4925baeed350a790e8fd6e7798c3e6ccab3821b60d961430fa48764968"}
Oct 31 13:29:13.903147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.903139    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68" event={"ID":"6ae32076-375d-48d2-8640-4e9ee3913caf","Type":"ContainerStarted","Data":"ed10358ef7c1f92beb7fed76bec79c2ca684c42b09ed53e39800a02f4d7d5040"}
Oct 31 13:29:13.905601 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.905578    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"]
Oct 31 13:29:13.910030 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.910011    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:13.938223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.936921    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"]
Oct 31 13:29:13.947537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.947308    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68" podStartSLOduration=1.947293262 podStartE2EDuration="1.947293262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:13.928958821 +0000 UTC m=+1551.327307447" watchObservedRunningTime="2026-10-31 13:29:13.947293262 +0000 UTC m=+1551.345641886"
Oct 31 13:29:13.952702 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.949084    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"]
Oct 31 13:29:13.952702 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.949192    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:13.967455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:13.967422    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"]
Oct 31 13:29:14.012489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.012459    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:14.021163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.018799    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"]
Oct 31 13:29:14.023566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.023405    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.023566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.023410    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.023566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.023478    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.023566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.023508    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.023851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.023572    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.023851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.023597    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bktkl\" (UniqueName: \"kubernetes.io/projected/b8571392-d978-4512-939b-69b9fadb6734-kube-api-access-bktkl\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.023851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.023633    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8571392-d978-4512-939b-69b9fadb6734-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.023851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.023675    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8571392-d978-4512-939b-69b9fadb6734-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.023851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.023722    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.023851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.023755    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.023851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.023850    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.024264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.023891    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp4kb\" (UniqueName: \"kubernetes.io/projected/056cecdf-d4f4-44c9-8cc5-416e24b29f86-kube-api-access-wp4kb\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.024264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.023946    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.024264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.023975    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.024264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.024250    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.024473 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.024295    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8571392-d978-4512-939b-69b9fadb6734-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.025205 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.024930    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.025205 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.024966    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.027598 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.026596    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.043384 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.040885    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"]
Oct 31 13:29:14.128921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.128892    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.129106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129085    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxd5v\" (UniqueName: \"kubernetes.io/projected/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-kube-api-access-dxd5v\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.129189 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129132    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.129245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129190    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.129245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129237    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.129534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129267    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.129534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129296    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.129534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129356    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.129534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129384    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.129534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129407    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.129534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129448    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.129534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129477    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bktkl\" (UniqueName: \"kubernetes.io/projected/b8571392-d978-4512-939b-69b9fadb6734-kube-api-access-bktkl\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.129534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129517    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8571392-d978-4512-939b-69b9fadb6734-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.130042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129564    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8571392-d978-4512-939b-69b9fadb6734-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.130042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129616    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.130042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129654    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.130042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129723    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.130042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129772    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.130042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129800    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp4kb\" (UniqueName: \"kubernetes.io/projected/056cecdf-d4f4-44c9-8cc5-416e24b29f86-kube-api-access-wp4kb\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.130042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129857    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.130042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129908    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.130042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.129948    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.130042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.130007    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.130042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.130036    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8571392-d978-4512-939b-69b9fadb6734-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.130611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.130093    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.130611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.130119    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.130611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.130150    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.130810 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.130640    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.132329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.131372    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.132329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.131384    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.132329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.131637    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.132329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.132212    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.132690 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.132664    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8571392-d978-4512-939b-69b9fadb6734-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.132975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.132948    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.133329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.133311    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.133995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.133951    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.134236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.134202    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.135085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.135060    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.136142 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.136102    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8571392-d978-4512-939b-69b9fadb6734-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.137465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.137445    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.138035 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.137995    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.145541 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.145359    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bktkl\" (UniqueName: \"kubernetes.io/projected/b8571392-d978-4512-939b-69b9fadb6734-kube-api-access-bktkl\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.145541 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.145499    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp4kb\" (UniqueName: \"kubernetes.io/projected/056cecdf-d4f4-44c9-8cc5-416e24b29f86-kube-api-access-wp4kb\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.145830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.145806    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.146211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.146192    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8571392-d978-4512-939b-69b9fadb6734-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.212220 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:29:14.208560    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e04722_598c_4608_9742_440e327e389a.slice/crio-7b340b823bad6d5d700a1b1e5038c1d2ccaa8770424113065ee01990f70bb27c WatchSource:0}: Error finding container 7b340b823bad6d5d700a1b1e5038c1d2ccaa8770424113065ee01990f70bb27c: Status 404 returned error can't find the container with id 7b340b823bad6d5d700a1b1e5038c1d2ccaa8770424113065ee01990f70bb27c
Oct 31 13:29:14.212220 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.209331    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"]
Oct 31 13:29:14.214465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.214434    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:14.214555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.214515    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:14.214555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.214544    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:14.231258 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.231222    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.231347 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.231280    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxd5v\" (UniqueName: \"kubernetes.io/projected/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-kube-api-access-dxd5v\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.231347 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.231305    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.231469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.231346    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.231469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.231390    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.231469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.231415    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.231469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.231439    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.231682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.231552    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.231682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.231625    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.232764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.232727    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:14.233022 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.232999    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.233373 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.233334    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.233462 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.233365    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.233522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.233468    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.233623 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.233598    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.234915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.234842    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.235231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.235209    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.244153 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.244128    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.244331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.244316    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxd5v\" (UniqueName: \"kubernetes.io/projected/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-kube-api-access-dxd5v\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.282804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.282771    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:14.386228 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.386200    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"]
Oct 31 13:29:14.387524 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:29:14.387488    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5e815e_eb38_40c4_9651_e4dbbb92dc36.slice/crio-7f72ce4e9b19f8f78a9ce5b76c45303e94b99074cb94288e0b0ff6e1e12d535c WatchSource:0}: Error finding container 7f72ce4e9b19f8f78a9ce5b76c45303e94b99074cb94288e0b0ff6e1e12d535c: Status 404 returned error can't find the container with id 7f72ce4e9b19f8f78a9ce5b76c45303e94b99074cb94288e0b0ff6e1e12d535c
Oct 31 13:29:14.389750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.389553    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:14.391443 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.390648    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:14.391443 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.390717    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:14.391443 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.390757    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:14.545311 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.545285    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"]
Oct 31 13:29:14.547076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.547037    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:14.547173 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.547123    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:14.547173 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.547167    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:14.598548 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.598502    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"]
Oct 31 13:29:14.628977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.628945    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:14.629082 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.629025    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:14.629082 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.629069    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:14.766210 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.766150    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:14.766387 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.766243    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:14.766387 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.766281    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:14.770031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.768757    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"]
Oct 31 13:29:14.818184 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:14.818145    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:15.011605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.010267    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" event={"ID":"b8571392-d978-4512-939b-69b9fadb6734","Type":"ContainerStarted","Data":"663b7fae31d3997e9a3aa2e3842bac417d1b7aa1223a719ba0603aa3645d1645"}
Oct 31 13:29:15.011605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.010336    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" event={"ID":"b8571392-d978-4512-939b-69b9fadb6734","Type":"ContainerStarted","Data":"bdcb897decb9197fd920fd74b796639502605b2ede2de4c60907d2373ca587bb"}
Oct 31 13:29:15.023680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.023645    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" event={"ID":"170b8b28-9f7f-494f-b6e6-86d5af6dcfec","Type":"ContainerStarted","Data":"b19ba9d5b62e9322241e20971093865c63ece300d25bce3f12375a3c70ea2edf"}
Oct 31 13:29:15.081739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.062082    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" event={"ID":"056cecdf-d4f4-44c9-8cc5-416e24b29f86","Type":"ContainerStarted","Data":"0d1e14884122ce4152cf1a4fd94da9d5fd81fd968dda9b48d5baab35a82e7cd8"}
Oct 31 13:29:15.081739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.065029    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" event={"ID":"056cecdf-d4f4-44c9-8cc5-416e24b29f86","Type":"ContainerStarted","Data":"5853477e2f558eb5fa5b6883b371a6ae4e9c6b3fd68cb5ea54c0c32ac28dbda6"}
Oct 31 13:29:15.081739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.063635    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" podStartSLOduration=2.063615253 podStartE2EDuration="2.063615253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:15.060941677 +0000 UTC m=+1552.459290305" watchObservedRunningTime="2026-10-31 13:29:15.063615253 +0000 UTC m=+1552.461963878"
Oct 31 13:29:15.081739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.069803    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn" event={"ID":"69e04722-598c-4608-9742-440e327e389a","Type":"ContainerStarted","Data":"3a3470b0c35f73c4ef5c10cbe6cadf696674bebc1230b9fe1d147f3791082c1f"}
Oct 31 13:29:15.081739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.069854    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn" event={"ID":"69e04722-598c-4608-9742-440e327e389a","Type":"ContainerStarted","Data":"7b340b823bad6d5d700a1b1e5038c1d2ccaa8770424113065ee01990f70bb27c"}
Oct 31 13:29:15.081739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.072915    4428 generic.go:356] "Generic (PLEG): container finished" podID="6ae32076-375d-48d2-8640-4e9ee3913caf" containerID="353a7e4925baeed350a790e8fd6e7798c3e6ccab3821b60d961430fa48764968" exitCode=0
Oct 31 13:29:15.081739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.072997    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68" event={"ID":"6ae32076-375d-48d2-8640-4e9ee3913caf","Type":"ContainerDied","Data":"353a7e4925baeed350a790e8fd6e7798c3e6ccab3821b60d961430fa48764968"}
Oct 31 13:29:15.081739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.080265    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" event={"ID":"af5e815e-eb38-40c4-9651-e4dbbb92dc36","Type":"ContainerStarted","Data":"3c0f0842e7377dd501d6d955f7aab95cbba47bf1f6a3ea579bad79d33b4b773a"}
Oct 31 13:29:15.081739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.080318    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" event={"ID":"af5e815e-eb38-40c4-9651-e4dbbb92dc36","Type":"ContainerStarted","Data":"7f72ce4e9b19f8f78a9ce5b76c45303e94b99074cb94288e0b0ff6e1e12d535c"}
Oct 31 13:29:15.179553 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.179488    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" podStartSLOduration=2.1794658 podStartE2EDuration="2.1794658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:15.103519511 +0000 UTC m=+1552.501868135" watchObservedRunningTime="2026-10-31 13:29:15.1794658 +0000 UTC m=+1552.577814426"
Oct 31 13:29:15.201778 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.201545    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" podStartSLOduration=2.201526569 podStartE2EDuration="2.201526569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:15.142147409 +0000 UTC m=+1552.540496034" watchObservedRunningTime="2026-10-31 13:29:15.201526569 +0000 UTC m=+1552.599875193"
Oct 31 13:29:15.206261 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.206209    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn" podStartSLOduration=2.206192442 podStartE2EDuration="2.206192442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:15.17522161 +0000 UTC m=+1552.573570231" watchObservedRunningTime="2026-10-31 13:29:15.206192442 +0000 UTC m=+1552.604541067"
Oct 31 13:29:15.311961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.310842    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:15.332939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.329420    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:15.332939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.331151    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:15.332939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.331207    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 13:29:15.332939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.332076    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:15.332939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.332119    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 13:29:15.914894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.913939    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:15.935895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.929037    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:15.935895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:15.929151    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn" podUID="69e04722-598c-4608-9742-440e327e389a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 13:29:16.028924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:16.025220    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:16.040893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:16.036036    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:16.040893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:16.036116    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" podUID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 13:29:16.138892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:16.135537    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" event={"ID":"170b8b28-9f7f-494f-b6e6-86d5af6dcfec","Type":"ContainerStarted","Data":"144cde08a76993f7629edf6d1db20383968cf953baba73f8cb83aee2a1bc7146"}
Oct 31 13:29:16.193580 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:16.193524    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" podStartSLOduration=3.193507541 podStartE2EDuration="3.193507541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:16.189218145 +0000 UTC m=+1553.587566774" watchObservedRunningTime="2026-10-31 13:29:16.193507541 +0000 UTC m=+1553.591856166"
Oct 31 13:29:16.242277 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:16.242242    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:16.246988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:16.246951    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 13:29:16.305894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:16.294431    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:16.305894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:16.294509    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 13:29:16.413206 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:16.413133    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:16.415942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:16.415914    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:16.416179 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:16.416156    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" podUID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 13:29:17.024893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.014983    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:17.028027 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.027954    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:17.031899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.029396    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:17.031899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.029445    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn" podUID="69e04722-598c-4608-9742-440e327e389a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 13:29:17.058920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.058644    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:17.058920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.058699    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" podUID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 13:29:17.132894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.122032    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:29:17.283951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.282105    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:29:17.283951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.283336    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:17.283951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.283372    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 13:29:17.283951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.283583    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:17.283951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.283611    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 13:29:17.420214 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.418809    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:17.420214 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.418894    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" podUID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 13:29:17.420214 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.418965    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:17.543949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.538071    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:29:17.590029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.589472    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:17.624421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.610949    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:17.711186 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.711081    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:17.711656 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.711548    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:17.920951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.915164    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"
Oct 31 13:29:17.957893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.957110    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:17.957893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:17.957168    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn" podUID="69e04722-598c-4608-9742-440e327e389a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 13:29:18.074893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.028615    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:18.074893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.028676    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" podUID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 13:29:18.074893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.029497    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae32076-375d-48d2-8640-4e9ee3913caf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae32076-375d-48d2-8640-4e9ee3913caf-maas-api-openshift-ca\") pod \"6ae32076-375d-48d2-8640-4e9ee3913caf\" (UID: \"6ae32076-375d-48d2-8640-4e9ee3913caf\") "
Oct 31 13:29:18.074893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.029646    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ae32076-375d-48d2-8640-4e9ee3913caf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae32076-375d-48d2-8640-4e9ee3913caf-maas-api-xks-ca\") pod \"6ae32076-375d-48d2-8640-4e9ee3913caf\" (UID: \"6ae32076-375d-48d2-8640-4e9ee3913caf\") "
Oct 31 13:29:18.074893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.029888    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae32076-375d-48d2-8640-4e9ee3913caf-kube-api-access-th2hc\" (UniqueName: \"kubernetes.io/projected/6ae32076-375d-48d2-8640-4e9ee3913caf-kube-api-access-th2hc\") pod \"6ae32076-375d-48d2-8640-4e9ee3913caf\" (UID: \"6ae32076-375d-48d2-8640-4e9ee3913caf\") "
Oct 31 13:29:18.119135 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.119084    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae32076-375d-48d2-8640-4e9ee3913caf-maas-api-openshift-ca" pod "6ae32076-375d-48d2-8640-4e9ee3913caf" (UID: "6ae32076-375d-48d2-8640-4e9ee3913caf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:29:18.221193 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.221158    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae32076-375d-48d2-8640-4e9ee3913caf-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:18.249307 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.249262    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:29:18.278923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.271139    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded"
Oct 31 13:29:18.283949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.282684    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae32076-375d-48d2-8640-4e9ee3913caf-maas-api-xks-ca" pod "6ae32076-375d-48d2-8640-4e9ee3913caf" (UID: "6ae32076-375d-48d2-8640-4e9ee3913caf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:29:18.283949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.283167    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:29:18.322893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.322162    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:29:18.352895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.323228    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:29:18.352895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.332497    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:18.352895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.332584    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 13:29:18.408898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.401245    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:18.428895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.425537    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 13:29:18.429358 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.429322    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:18.438188 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.438150    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:29:18.440081 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.440056    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:29:18.453402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.453309    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:29:18.810709 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.810657    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:18.838943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.560581    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae32076-375d-48d2-8640-4e9ee3913caf-kube-api-access-th2hc" pod "6ae32076-375d-48d2-8640-4e9ee3913caf" (UID: "6ae32076-375d-48d2-8640-4e9ee3913caf"). InnerVolumeSpecName "kube-api-access-th2hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:29:18.840339 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.840304    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:18.964936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:18.919217    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:29:19.063147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.063039    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:29:19.106908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.106848    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:19.107432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.106940    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn" podUID="69e04722-598c-4608-9742-440e327e389a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 13:29:19.110900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.108211    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae32076-375d-48d2-8640-4e9ee3913caf-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:19.110900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.108357    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:29:19.119894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.117934    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:19.220166 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.220129    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:19.535540 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.535496    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" podUID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 13:29:19.585047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.459367    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:29:19.585047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.511680    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:19.585047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.512760    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:29:19.585047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.259192    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:29:19.585047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.536717    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:29:19.585047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.536758    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:29:19.585047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.233719    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:19.585047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.537230    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" podUID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 13:29:19.611905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.297932    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:19.611905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.597332    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 13:29:19.611905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.337746    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:19.611905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.597403    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 13:29:19.611905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.611631    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:19.611905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.611709    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" podUID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 13:29:19.815728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.751556    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th2hc\" (UniqueName: \"kubernetes.io/projected/6ae32076-375d-48d2-8640-4e9ee3913caf-kube-api-access-th2hc\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:19.823887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:19.777145    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:20.125730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.120846    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:20.125730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.120927    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn" podUID="69e04722-598c-4608-9742-440e327e389a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 13:29:20.125730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.120997    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:29:20.242199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.241525    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:20.242199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.241608    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" podUID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 13:29:20.266895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.261636    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:29:20.315901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.303450    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:20.345986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.345936    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:29:20.355930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.354462    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:20.355930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.354515    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 13:29:20.380037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.359524    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:29:20.450830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.450776    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:20.451273 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.451251    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:20.451450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.451428    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 13:29:20.471249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.471210    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:20.471727 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.471699    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47" podUID="ea978050-20b0-4148-ab41-75844fe1b637" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:20.502196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.502166    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:20.502467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.502444    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" podUID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 13:29:20.535964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.535916    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59" podUID="cd0ac048-d186-4f3c-929d-eb870afbc801" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:20.599990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.570015    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:20.600213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.570052    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:29:20.626911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.570073    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:29:20.837904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.827490    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:20.916889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.907800    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"
Oct 31 13:29:20.947912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.940484    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:20.947912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:20.940551    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn" podUID="69e04722-598c-4608-9742-440e327e389a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 13:29:21.165949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.152114    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:21.165949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.152187    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" podUID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 13:29:21.165949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.152897    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:21.309988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.294792    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:29:21.309988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.308579    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:29:21.337748 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.336985    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:21.367110 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.348921    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:21.379487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.378487    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:21.379487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.378548    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 13:29:21.418095 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.418009    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:21.418327 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.418300    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" podUID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 13:29:21.460772 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.460737    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:21.461087 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.461054    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 13:29:21.467469 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:29:21.467435    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:29:21.747799 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.656004    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:21.747799 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.656046    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:21.747799 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.656067    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:29:21.747799 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.666611    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68" event={"ID":"6ae32076-375d-48d2-8640-4e9ee3913caf","Type":"ContainerDied","Data":"ed10358ef7c1f92beb7fed76bec79c2ca684c42b09ed53e39800a02f4d7d5040"}
Oct 31 13:29:21.747799 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.742192    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed10358ef7c1f92beb7fed76bec79c2ca684c42b09ed53e39800a02f4d7d5040"
Oct 31 13:29:21.844902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.841841    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:29:21.992893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:21.984343    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:22.079965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:22.072228    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:22.102128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:22.102085    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" podUID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 13:29:22.189893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:22.164820    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:22.190804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:22.167730    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:22.191029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:22.190998    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn" podUID="69e04722-598c-4608-9742-440e327e389a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 13:29:22.245891 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:29:22.232079    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae32076_375d_48d2_8640_4e9ee3913caf.slice/crio-ed10358ef7c1f92beb7fed76bec79c2ca684c42b09ed53e39800a02f4d7d5040\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae32076_375d_48d2_8640_4e9ee3913caf.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:29:22.438954 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:29:22.405977    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae32076_375d_48d2_8640_4e9ee3913caf.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae32076_375d_48d2_8640_4e9ee3913caf.slice/crio-ed10358ef7c1f92beb7fed76bec79c2ca684c42b09ed53e39800a02f4d7d5040\": RecentStats: unable to find data in memory cache]"
Oct 31 13:29:22.570893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:22.425577    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:22.570893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:22.442148    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:22.570893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:22.442208    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:22.578075 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:22.497887    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:29:22.605923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:22.602708    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 13:29:22.605923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:22.567170    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 13:29:22.606578 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:22.507256    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:22.615447 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:22.615386    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" podUID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 13:29:22.944099 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:22.944032    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:29:23.145620 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.145490    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:29:23.146261 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.146234    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:23.438902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.429215    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:29:23.470070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.438773    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:23.470368 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.470332    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" podUID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 13:29:23.470505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.453557    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:29:23.531906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.509850    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:23.531906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.146447    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn" podUID="69e04722-598c-4608-9742-440e327e389a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 13:29:23.595533 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.595499    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:23.619893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.613236    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:23.619893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.618005    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:23.619893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.618048    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" podUID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 13:29:23.622497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.622473    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:29:23.623794 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.623771    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:23.662924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.639950    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:23.662924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.639998    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 13:29:23.662924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.640444    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:23.662924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.640481    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 13:29:23.904376 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.904023    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 13:29:23.904376 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.904086    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn" podUID="69e04722-598c-4608-9742-440e327e389a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 13:29:23.961908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:23.960222    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:29:24.063901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.046072    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:24.063901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.046127    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" podUID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 13:29:24.254921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.244960    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:24.254921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.245017    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 13:29:24.284139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.284109    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:24.284434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.284409    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 13:29:24.452892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.423800    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:24.452892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.423884    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" podUID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 13:29:24.476394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.454634    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:24.476394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.454732    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:29:24.476394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.472440    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:24.516900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.516774    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:24.643890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.637604    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:29:24.655837 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.655691    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:24.690891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.671457    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:24.777709 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.777618    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:29:24.945893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.944120    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:29:24.969242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:24.964992    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:25.033948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:25.021099    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:29:25.045387 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:25.045355    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 13:29:25.052594 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:25.052564    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" podUID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 13:29:25.215075 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:25.215046    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:29:25.303727 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:25.303102    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:29:25.316639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:25.314811    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 13:29:25.317020 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:25.316988    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 13:29:25.384167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:25.384135    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 13:29:25.384459 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:25.384431    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 13:29:25.393917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:25.393437    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 13:29:25.393917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:25.393494    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" podUID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 13:29:25.423732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:25.423664    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:29:25.544903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:25.520542    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:25.782644 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:25.782615    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:25.784704 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:25.783917    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:25.788165 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:25.788116    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:29:25.861958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:25.861037    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:29:26.046486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:26.046412    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:26.048104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:26.048017    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:26.049356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:26.049327    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:29:26.624688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:26.622126    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:26.624688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:26.623085    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:26.627205 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:26.625321    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:26.627205 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:26.625927    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:26.632422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:26.628953    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:29:26.632422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:26.630287    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:29:26.954528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:26.954455    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:26.955063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:26.955039    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:26.957286 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:26.957264    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:29:28.614296 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.614264    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:28.829814 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.829770    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"]
Oct 31 13:29:28.830533 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.830512    4428 cpu_manager.go:414] "containerMap: removing container" podUID="6ae32076-375d-48d2-8640-4e9ee3913caf" containerName="revoke-keys"
Oct 31 13:29:28.830533 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.830534    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae32076-375d-48d2-8640-4e9ee3913caf" containerName="revoke-keys"
Oct 31 13:29:28.830742 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.830729    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae32076-375d-48d2-8640-4e9ee3913caf" containerName="revoke-keys"
Oct 31 13:29:28.836539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.836507    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.856322 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.856294    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"]
Oct 31 13:29:28.869571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.869492    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.869571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.869535    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hh67\" (UniqueName: \"kubernetes.io/projected/095761dd-4e84-4051-b472-b85d43d444c9-kube-api-access-9hh67\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.869760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.869576    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095761dd-4e84-4051-b472-b85d43d444c9-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.869760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.869611    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095761dd-4e84-4051-b472-b85d43d444c9-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.869760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.869666    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.869760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.869692    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095761dd-4e84-4051-b472-b85d43d444c9-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.870034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.869985    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.870196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.870173    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.870310 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.870292    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.930571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.930548    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"]
Oct 31 13:29:28.934959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.934939    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:28.948515 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.948479    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"]
Oct 31 13:29:28.971434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.971407    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.971434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.971436    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095761dd-4e84-4051-b472-b85d43d444c9-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.971648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.971493    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8s4b\" (UniqueName: \"kubernetes.io/projected/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-kube-api-access-m8s4b\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:28.971648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.971540    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.971648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.971595    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:28.971819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.971801    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:28.971950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.971927    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:28.972011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.971978    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:28.972120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.972095    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.972196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.972144    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:28.972196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.972186    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.972301 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.972210    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:28.972301 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.972257    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095761dd-4e84-4051-b472-b85d43d444c9-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.972494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.972473    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.972544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.972491    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:28.972587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.972565    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.972684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.972666    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:28.972769 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.972701    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.972769 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.972726    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hh67\" (UniqueName: \"kubernetes.io/projected/095761dd-4e84-4051-b472-b85d43d444c9-kube-api-access-9hh67\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.972769 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.972764    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095761dd-4e84-4051-b472-b85d43d444c9-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.972969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.972802    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095761dd-4e84-4051-b472-b85d43d444c9-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.972969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.972898    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.973334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.973279    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.974278 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.974254    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.975778 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.975752    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095761dd-4e84-4051-b472-b85d43d444c9-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.989942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.989921    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hh67\" (UniqueName: \"kubernetes.io/projected/095761dd-4e84-4051-b472-b85d43d444c9-kube-api-access-9hh67\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:28.992058 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:28.992026    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095761dd-4e84-4051-b472-b85d43d444c9-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:29.032711 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.032676    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"]
Oct 31 13:29:29.039634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.039602    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.050624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.050587    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"]
Oct 31 13:29:29.074269 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.074238    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12211035-140d-4934-9c27-36ef18fd6d1b-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.074475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.074291    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.074475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.074360    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.074475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.074409    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.074475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.074444    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg27w\" (UniqueName: \"kubernetes.io/projected/12211035-140d-4934-9c27-36ef18fd6d1b-kube-api-access-wg27w\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.074475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.074466    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.074760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.074495    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.074760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.074523    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12211035-140d-4934-9c27-36ef18fd6d1b-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.074760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.074554    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.074760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.074578    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.074760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.074598    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.075046 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.074955    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.075370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.075267    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.075370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.075295    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.075370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.075319    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.075607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.075413    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12211035-140d-4934-9c27-36ef18fd6d1b-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.075607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.075469    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8s4b\" (UniqueName: \"kubernetes.io/projected/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-kube-api-access-m8s4b\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.075607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.075534    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.075766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.075697    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.075827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.075792    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.076151 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.076116    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.076260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.076202    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.076397 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.076375    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.078065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.078042    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.079797 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.079770    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.087720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.087696    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8s4b\" (UniqueName: \"kubernetes.io/projected/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-kube-api-access-m8s4b\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.087937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.087917    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.160701 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.160580    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:29.178909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.177277    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.178909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.177332    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.178909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.177428    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.178909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.177493    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12211035-140d-4934-9c27-36ef18fd6d1b-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.178909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.177561    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.178909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.177707    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12211035-140d-4934-9c27-36ef18fd6d1b-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.178909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.177779    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg27w\" (UniqueName: \"kubernetes.io/projected/12211035-140d-4934-9c27-36ef18fd6d1b-kube-api-access-wg27w\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.178909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.177811    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.178909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.177882    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12211035-140d-4934-9c27-36ef18fd6d1b-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.178909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.177964    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.178909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.178403    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.178909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.178558    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.178909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.178678    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12211035-140d-4934-9c27-36ef18fd6d1b-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.179645 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.178952    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.182129 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.182101    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.182305 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.182240    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12211035-140d-4934-9c27-36ef18fd6d1b-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.190965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.190938    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12211035-140d-4934-9c27-36ef18fd6d1b-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.191483 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.191461    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg27w\" (UniqueName: \"kubernetes.io/projected/12211035-140d-4934-9c27-36ef18fd6d1b-kube-api-access-wg27w\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.251512 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.251483    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:29.368299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.368269    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:29.465930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.465901    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"]
Oct 31 13:29:29.466752 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:29:29.466722    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095761dd_4e84_4051_b472_b85d43d444c9.slice/crio-afc2d23b1e6a9c7f115f8514138f3b9047a638e27b71ccd7560fd04c5f8e00a5 WatchSource:0}: Error finding container afc2d23b1e6a9c7f115f8514138f3b9047a638e27b71ccd7560fd04c5f8e00a5: Status 404 returned error can't find the container with id afc2d23b1e6a9c7f115f8514138f3b9047a638e27b71ccd7560fd04c5f8e00a5
Oct 31 13:29:29.469715 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.469685    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:29.469796 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.469750    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:29.469796 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.469779    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:29.594887 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:29:29.591782    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a8d6f8_00bd_42b5_8eb8_3b5053cf2e8c.slice/crio-0c9c22f1e261e3e48408dd439aa2a30e0d2f897f13c16d3234bdd430131133f2 WatchSource:0}: Error finding container 0c9c22f1e261e3e48408dd439aa2a30e0d2f897f13c16d3234bdd430131133f2: Status 404 returned error can't find the container with id 0c9c22f1e261e3e48408dd439aa2a30e0d2f897f13c16d3234bdd430131133f2
Oct 31 13:29:29.603498 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.603468    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"]
Oct 31 13:29:29.606689 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.606577    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:29.606826 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.606793    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:29.606983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.606936    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:29.688237 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.688204    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"]
Oct 31 13:29:29.690169 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:29:29.690139    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12211035_140d_4934_9c27_36ef18fd6d1b.slice/crio-92d1dbc2818fc925d0ec7c61887f80c34ca7a95bf64fa7e5d18715f474497e66 WatchSource:0}: Error finding container 92d1dbc2818fc925d0ec7c61887f80c34ca7a95bf64fa7e5d18715f474497e66: Status 404 returned error can't find the container with id 92d1dbc2818fc925d0ec7c61887f80c34ca7a95bf64fa7e5d18715f474497e66
Oct 31 13:29:29.693423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.693390    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:29.693545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.693467    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:29.693545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.693514    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:29.899230 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.899187    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" event={"ID":"12211035-140d-4934-9c27-36ef18fd6d1b","Type":"ContainerStarted","Data":"5291084f9d7bdedbd61f2509b5e459d37a540487afbd35f270dfcdf434410aee"}
Oct 31 13:29:29.899406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.899266    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" event={"ID":"12211035-140d-4934-9c27-36ef18fd6d1b","Type":"ContainerStarted","Data":"92d1dbc2818fc925d0ec7c61887f80c34ca7a95bf64fa7e5d18715f474497e66"}
Oct 31 13:29:29.903918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.902275    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" event={"ID":"095761dd-4e84-4051-b472-b85d43d444c9","Type":"ContainerStarted","Data":"05e998ce72c6c5ba640a86174ae695bf5a17360f8e8a63ad64a1951932ab5fa4"}
Oct 31 13:29:29.903918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.902334    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" event={"ID":"095761dd-4e84-4051-b472-b85d43d444c9","Type":"ContainerStarted","Data":"afc2d23b1e6a9c7f115f8514138f3b9047a638e27b71ccd7560fd04c5f8e00a5"}
Oct 31 13:29:29.933132 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.933094    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" event={"ID":"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c","Type":"ContainerStarted","Data":"776373b78bb26d3d9e1d16a0a0aac2ac43c7a172a2f4680260027487b4f7720c"}
Oct 31 13:29:29.933348 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.933156    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" event={"ID":"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c","Type":"ContainerStarted","Data":"0c9c22f1e261e3e48408dd439aa2a30e0d2f897f13c16d3234bdd430131133f2"}
Oct 31 13:29:29.935250 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.935199    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podStartSLOduration=0.935178379 podStartE2EDuration="935.178379ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:29.924458224 +0000 UTC m=+1567.322806849" watchObservedRunningTime="2026-10-31 13:29:29.935178379 +0000 UTC m=+1567.333527026"
Oct 31 13:29:29.960123 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.960025    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podStartSLOduration=1.960007127 podStartE2EDuration="1.960007127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:29.957571034 +0000 UTC m=+1567.355919658" watchObservedRunningTime="2026-10-31 13:29:29.960007127 +0000 UTC m=+1567.358355753"
Oct 31 13:29:29.986440 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:29.986363    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podStartSLOduration=1.986349929 podStartE2EDuration="1.986349929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:29.982368678 +0000 UTC m=+1567.380717339" watchObservedRunningTime="2026-10-31 13:29:29.986349929 +0000 UTC m=+1567.384698552"
Oct 31 13:29:30.160818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:30.160784    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:30.162257 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:30.162227    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:30.162399 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:30.162272    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:30.253138 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:30.253048    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:30.254995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:30.254963    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:30.255236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:30.255203    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:30.380969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:30.380936    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:30.404889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:30.381086    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:30.426004 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:30.385606    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:30.426360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:30.426324    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:31.240205 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.240167    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:31.240753 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.240716    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:31.396127 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.396080    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:31.396423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.396393    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:31.426896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.426835    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:31.427156 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.427128    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:31.440991 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.440957    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"]
Oct 31 13:29:31.494397 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.494315    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"]
Oct 31 13:29:31.525853 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.494688    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"
Oct 31 13:29:31.537562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.536138    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:29:31.537562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.536361    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:29:31.672906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.664045    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbwt4\" (UniqueName: \"kubernetes.io/projected/26dd467a-54e2-4d31-a82e-74fcf4f39120-kube-api-access-dbwt4\") pod \"maas-api-revoke-sub-768a7056-zg5j4\" (UID: \"26dd467a-54e2-4d31-a82e-74fcf4f39120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"
Oct 31 13:29:31.672906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.664121    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26dd467a-54e2-4d31-a82e-74fcf4f39120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768a7056-zg5j4\" (UID: \"26dd467a-54e2-4d31-a82e-74fcf4f39120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"
Oct 31 13:29:31.672906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.664156    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26dd467a-54e2-4d31-a82e-74fcf4f39120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768a7056-zg5j4\" (UID: \"26dd467a-54e2-4d31-a82e-74fcf4f39120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"
Oct 31 13:29:31.937036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.810069    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbwt4\" (UniqueName: \"kubernetes.io/projected/26dd467a-54e2-4d31-a82e-74fcf4f39120-kube-api-access-dbwt4\") pod \"maas-api-revoke-sub-768a7056-zg5j4\" (UID: \"26dd467a-54e2-4d31-a82e-74fcf4f39120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"
Oct 31 13:29:31.937036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.929023    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26dd467a-54e2-4d31-a82e-74fcf4f39120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768a7056-zg5j4\" (UID: \"26dd467a-54e2-4d31-a82e-74fcf4f39120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"
Oct 31 13:29:31.937036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.929071    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26dd467a-54e2-4d31-a82e-74fcf4f39120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768a7056-zg5j4\" (UID: \"26dd467a-54e2-4d31-a82e-74fcf4f39120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"
Oct 31 13:29:31.937036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.810369    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:31.937036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:31.931546    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26dd467a-54e2-4d31-a82e-74fcf4f39120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768a7056-zg5j4\" (UID: \"26dd467a-54e2-4d31-a82e-74fcf4f39120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"
Oct 31 13:29:32.048896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:32.039324    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26dd467a-54e2-4d31-a82e-74fcf4f39120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768a7056-zg5j4\" (UID: \"26dd467a-54e2-4d31-a82e-74fcf4f39120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"
Oct 31 13:29:32.139823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:32.139786    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbwt4\" (UniqueName: \"kubernetes.io/projected/26dd467a-54e2-4d31-a82e-74fcf4f39120-kube-api-access-dbwt4\") pod \"maas-api-revoke-sub-768a7056-zg5j4\" (UID: \"26dd467a-54e2-4d31-a82e-74fcf4f39120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"
Oct 31 13:29:32.178904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:32.177001    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:32.178904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:32.177064    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:32.261707 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:32.261674    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:32.262285 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:32.262258    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:32.262941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:32.262920    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"
Oct 31 13:29:32.372903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:32.370999    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:32.372903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:32.371058    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:32.936420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:32.936386    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:33.175890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:33.171968    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:33.175890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:33.172069    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:33.283949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:33.270366    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:33.283949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:33.270428    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:33.449697 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:33.423439    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:33.449697 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:33.423495    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:33.564948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:33.559412    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:29:33.761063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:33.633978    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:29:33.761063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:33.754051    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:29:33.844743 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:33.832906    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:29:33.868122 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:33.846287    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:33.884232 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:33.875198    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:29:33.884232 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:33.875556    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Oct 31 13:29:33.891757 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:33.889289    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:33.915488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:33.915454    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"]
Oct 31 13:29:34.036909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:33.952471    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:34.170066 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:34.170022    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:34.170405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:34.170365    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:34.255924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:34.255850    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:34.256205 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:34.256169    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:34.369780 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:34.369697    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:34.375423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:34.375374    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:34.604025 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:34.603984    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:34.604906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:34.604884    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:29:34.673377 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:34.673213    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4" event={"ID":"26dd467a-54e2-4d31-a82e-74fcf4f39120","Type":"ContainerStarted","Data":"57440360ab62ae322a2a47724a874350de6cc0525f947f0874b0a8ea0a56b12c"}
Oct 31 13:29:34.836257 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:34.833425    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:34.836257 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:34.833955    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:29:34.890893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:34.886959    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:29:34.890893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:34.887337    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:29:34.890893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:34.888760    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:34.891885 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:34.891843    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:34.964080 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:34.958992    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:29:35.057906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.052162    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:29:35.057906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.052236    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:29:35.057906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.052276    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:29:35.057906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.052311    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:29:35.057906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.052345    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:29:35.057906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.052379    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:29:35.057906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.052414    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:29:35.192073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.192015    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:35.192367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.192334    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:35.296395 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.294990    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:35.296395 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.295046    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:35.372263 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.372231    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:35.372883 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.372833    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:35.685461 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.684775    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:35.723955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.720018    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Oct 31 13:29:35.917958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.848390    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:29:35.918906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.848433    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:29:35.922894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.903819    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:29:36.372141 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.903859    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:36.410896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.903904    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:36.957937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:36.953109    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:37.010898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:36.959465    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:29:37.514241 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:36.425180    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47" podUID="ea978050-20b0-4148-ab41-75844fe1b637" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:37.679125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:36.476433    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:37.757895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:37.726640    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:37.854969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:36.998821    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6" podUID="149a9b52-ef95-4569-b947-e6e14d707339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Oct 31 13:29:37.854969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:36.998902    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg" podUID="5c86b906-a276-4e67-8e58-79a17118f5fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Oct 31 13:29:37.854969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:37.123786    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59" podUID="cd0ac048-d186-4f3c-929d-eb870afbc801" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:37.854969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:35.903923    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:29:37.854969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:36.350983    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:29:37.883895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:37.387983    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:38.052902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.050059    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:29:38.052902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.050580    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:29:38.079900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:36.556257    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:38.079900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:37.283165    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:38.079900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.062507    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:38.101925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:37.284570    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:38.105941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.103848    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj" podUID="208fddf2-00ef-4f4a-944a-f3d93ba37911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:38.124888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:36.998773    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:29:38.138067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.138019    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-55fbbbf9dc-w9wjk" podUID="e328741d-31fd-40db-8cbb-8c9b518f4cde" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.39:50052\" within 1s: context deadline exceeded"
Oct 31 13:29:38.138311 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.097761    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:38.168831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.146280    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:38.168831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.146347    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:38.168831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.062591    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded"
Oct 31 13:29:38.168831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:37.839537    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:38.168831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.149129    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:38.178537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:37.408144    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:38.211144 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:29:38.179627    4428 logging.go:55] [core] [Channel #809 SubChannel #813]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.39:50052", ServerName: "10.134.0.39:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.39:50052: operation was canceled"
Oct 31 13:29:38.211144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.179713    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:38.211144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.179765    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:38.291034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.290982    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:38.291908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.291798    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:38.331812 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.331776    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:38.432217 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.432118    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:38.432458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.334638    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4" event={"ID":"26dd467a-54e2-4d31-a82e-74fcf4f39120","Type":"ContainerStarted","Data":"63dc0e408c496a7661acfcae49b6560e4edb77e9f161961568314c2ede6120cf"}
Oct 31 13:29:38.432636 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.379029    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:38.432761 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.432740    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:38.488324 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.476948    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:38.595895 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:29:38.589664    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26dd467a_54e2_4d31_a82e_74fcf4f39120.slice/crio-63dc0e408c496a7661acfcae49b6560e4edb77e9f161961568314c2ede6120cf.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 13:29:38.832974 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.750155    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:38.832974 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.750275    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded"
Oct 31 13:29:38.994901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.990007    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:29:38.994901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:38.990459    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:29:39.005027 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.004976    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:39.041901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.038042    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:29:39.070893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.069940    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:39.070893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.070722    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:39.140955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.137013    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:39.165988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.165944    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb" podUID="fb96175b-4567-4467-a2fa-1aacfeffa985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:39.166253 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.164656    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:39.173894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.169919    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:39.185894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.177977    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:39.204897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.180184    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:39.217896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.210489    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4" podStartSLOduration=8.210467255 podStartE2EDuration="8.210467255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:38.488180891 +0000 UTC m=+1575.886529527" watchObservedRunningTime="2026-10-31 13:29:39.210467255 +0000 UTC m=+1576.608815881"
Oct 31 13:29:39.259223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.259190    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:39.259486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.259459    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:39.389893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.384015    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:39.389893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.384090    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:39.389893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.389345    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:39.389893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.389399    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst" podUID="f5b5a82a-0de7-4c45-94a5-b8c046ea9387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:39.493704 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.485595    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:29:39.543916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.536115    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:39.543916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.536187    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6" podUID="37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:39.543916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.537408    4428 generic.go:356] "Generic (PLEG): container finished" podID="26dd467a-54e2-4d31-a82e-74fcf4f39120" containerID="63dc0e408c496a7661acfcae49b6560e4edb77e9f161961568314c2ede6120cf" exitCode=0
Oct 31 13:29:39.543916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.537462    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4" event={"ID":"26dd467a-54e2-4d31-a82e-74fcf4f39120","Type":"ContainerDied","Data":"63dc0e408c496a7661acfcae49b6560e4edb77e9f161961568314c2ede6120cf"}
Oct 31 13:29:39.698060 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.698018    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:39.698659 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:39.698628    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns" podUID="93314798-1f48-408a-b87c-d8f809d2f14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:40.016099 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.007963    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:29:40.016099 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.008368    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:29:40.053968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.049233    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:40.053968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.053088    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Oct 31 13:29:40.062024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.057948    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:40.062024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.058003    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2" podUID="9796198f-74dc-491b-b5fe-7ee7bb9ae809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:40.103155 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.081453    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:40.103155 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.081515    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl" podUID="7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:40.103155 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.081641    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:40.103155 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.082012    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:29:40.176898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.175327    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:40.176898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.175393    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:40.200895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.198982    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:29:40.208895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.206969    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:40.254895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.254057    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:40.254895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.254123    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:40.256151 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.256021    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:40.256151 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.256088    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl" podUID="a4853846-205d-4d62-a056-db336dee24cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:40.370404 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.370268    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:40.370404 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.370339    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:40.486896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.486104    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:29:40.712937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:40.712901    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:41.019401 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.016022    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.019401 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.016443    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded"
Oct 31 13:29:41.019401 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.016510    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.064853 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.064806    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:41.065196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.065142    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt" podUID="a64a3776-6565-48d0-8c36-5c4d979eeb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.106907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.087023    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.106907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.087461    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:29:41.106907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.092078    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.179895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.166704    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:41.179895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.166778    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:41.211894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.206238    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.231033 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.227735    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.256892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.255923    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:41.256892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.256010    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:41.369339 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.369247    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:41.391898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.377968    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:41.495893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.491666    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.605767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.605736    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:41.606064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.606035    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x" podUID="04337deb-7a21-4e58-8a41-098d8be2e918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:41.845893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.829363    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:41.845893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:41.829435    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn" podUID="69e04722-598c-4608-9742-440e327e389a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:42.038282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.037352    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:42.038282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.037750    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:29:42.096000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.094037    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:42.111895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.104008    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:42.182247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.115403    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:42.182937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.179725    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:42.183291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.183262    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" podUID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:42.215067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.209440    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:42.215067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.209499    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:42.215067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.213347    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:29:42.243314 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.241759    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:42.261575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.260050    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:42.261575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.260112    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:42.407445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.407332    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:42.407445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.407405    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:42.503324 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.503278    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:42.740922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.733314    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:42.740922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.733379    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:42.740922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.735774    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:42.740922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:42.735832    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:43.012893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.012824    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:43.014194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.014129    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" podUID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:43.045352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.045308    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:43.046923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.046893    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:29:43.095897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.066680    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"
Oct 31 13:29:43.104072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.104020    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:43.166147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.166111    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:43.166443 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.166414    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:43.196543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.196484    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:29:43.204932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.200664    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:29:43.223462 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.210461    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4" event={"ID":"26dd467a-54e2-4d31-a82e-74fcf4f39120","Type":"ContainerDied","Data":"57440360ab62ae322a2a47724a874350de6cc0525f947f0874b0a8ea0a56b12c"}
Oct 31 13:29:43.223767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.223744    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57440360ab62ae322a2a47724a874350de6cc0525f947f0874b0a8ea0a56b12c"
Oct 31 13:29:43.223932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.214192    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:43.266321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.265436    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:43.299903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.290013    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26dd467a-54e2-4d31-a82e-74fcf4f39120-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26dd467a-54e2-4d31-a82e-74fcf4f39120-maas-api-openshift-ca\") pod \"26dd467a-54e2-4d31-a82e-74fcf4f39120\" (UID: \"26dd467a-54e2-4d31-a82e-74fcf4f39120\") "
Oct 31 13:29:43.312071 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.290610    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26dd467a-54e2-4d31-a82e-74fcf4f39120-maas-api-openshift-ca" pod "26dd467a-54e2-4d31-a82e-74fcf4f39120" (UID: "26dd467a-54e2-4d31-a82e-74fcf4f39120"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:29:43.335575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.335532    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:43.335884 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.335837    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:43.336495 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.336468    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26dd467a-54e2-4d31-a82e-74fcf4f39120-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26dd467a-54e2-4d31-a82e-74fcf4f39120-maas-api-xks-ca\") pod \"26dd467a-54e2-4d31-a82e-74fcf4f39120\" (UID: \"26dd467a-54e2-4d31-a82e-74fcf4f39120\") "
Oct 31 13:29:43.343991 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.336698    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26dd467a-54e2-4d31-a82e-74fcf4f39120-kube-api-access-dbwt4\" (UniqueName: \"kubernetes.io/projected/26dd467a-54e2-4d31-a82e-74fcf4f39120-kube-api-access-dbwt4\") pod \"26dd467a-54e2-4d31-a82e-74fcf4f39120\" (UID: \"26dd467a-54e2-4d31-a82e-74fcf4f39120\") "
Oct 31 13:29:43.369702 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.344998    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26dd467a-54e2-4d31-a82e-74fcf4f39120-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:43.373152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.373063    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:43.373436 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.373394    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:43.509896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.493022    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26dd467a-54e2-4d31-a82e-74fcf4f39120-kube-api-access-dbwt4" pod "26dd467a-54e2-4d31-a82e-74fcf4f39120" (UID: "26dd467a-54e2-4d31-a82e-74fcf4f39120"). InnerVolumeSpecName "kube-api-access-dbwt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:29:43.510244 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.510202    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:43.598046 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.597957    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbwt4\" (UniqueName: \"kubernetes.io/projected/26dd467a-54e2-4d31-a82e-74fcf4f39120-kube-api-access-dbwt4\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:43.751976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.740088    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Oct 31 13:29:43.845611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.840120    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26dd467a-54e2-4d31-a82e-74fcf4f39120-maas-api-xks-ca" pod "26dd467a-54e2-4d31-a82e-74fcf4f39120" (UID: "26dd467a-54e2-4d31-a82e-74fcf4f39120"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:29:43.868973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.868411    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26dd467a-54e2-4d31-a82e-74fcf4f39120-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:43.924074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.905915    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"]
Oct 31 13:29:43.924074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.914277    4428 cpu_manager.go:414] "containerMap: removing container" podUID="26dd467a-54e2-4d31-a82e-74fcf4f39120" containerName="revoke-keys"
Oct 31 13:29:43.924074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.914305    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26dd467a-54e2-4d31-a82e-74fcf4f39120" containerName="revoke-keys"
Oct 31 13:29:43.924074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.914568    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="26dd467a-54e2-4d31-a82e-74fcf4f39120" containerName="revoke-keys"
Oct 31 13:29:43.962073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:43.943386    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"
Oct 31 13:29:44.047074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.047033    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:44.246929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.047584    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:29:44.246929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.104357    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:29:44.249168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.141003    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:44.249488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.168123    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:44.250609 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.168518    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:44.250885 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.250841    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:44.251038 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.200640    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:44.251460 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.200100    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221eeffa-0e13-4f2b-b071-954dc973142a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4dc5e8b-hdfvx\" (UID: \"221eeffa-0e13-4f2b-b071-954dc973142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"
Oct 31 13:29:44.251675 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.251640    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221eeffa-0e13-4f2b-b071-954dc973142a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4dc5e8b-hdfvx\" (UID: \"221eeffa-0e13-4f2b-b071-954dc973142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"
Oct 31 13:29:44.252064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.252035    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"
Oct 31 13:29:44.252200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.252183    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l6qt\" (UniqueName: \"kubernetes.io/projected/221eeffa-0e13-4f2b-b071-954dc973142a-kube-api-access-5l6qt\") pod \"maas-api-revoke-sub-f4dc5e8b-hdfvx\" (UID: \"221eeffa-0e13-4f2b-b071-954dc973142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"
Oct 31 13:29:44.259670 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.259594    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:44.265896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.206174    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:29:44.322956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.229365    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:44.322956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.260303    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:44.322956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.321814    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:44.322956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.266124    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:44.365426 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.365390    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l6qt\" (UniqueName: \"kubernetes.io/projected/221eeffa-0e13-4f2b-b071-954dc973142a-kube-api-access-5l6qt\") pod \"maas-api-revoke-sub-f4dc5e8b-hdfvx\" (UID: \"221eeffa-0e13-4f2b-b071-954dc973142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"
Oct 31 13:29:44.365682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.365661    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221eeffa-0e13-4f2b-b071-954dc973142a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4dc5e8b-hdfvx\" (UID: \"221eeffa-0e13-4f2b-b071-954dc973142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"
Oct 31 13:29:44.365771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.365722    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221eeffa-0e13-4f2b-b071-954dc973142a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4dc5e8b-hdfvx\" (UID: \"221eeffa-0e13-4f2b-b071-954dc973142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"
Oct 31 13:29:44.372776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.372751    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:44.385303 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.385269    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:44.385537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.385506    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:44.388089 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.388067    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221eeffa-0e13-4f2b-b071-954dc973142a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4dc5e8b-hdfvx\" (UID: \"221eeffa-0e13-4f2b-b071-954dc973142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"
Oct 31 13:29:44.434893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.420400    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"]
Oct 31 13:29:44.496362 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.496328    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l6qt\" (UniqueName: \"kubernetes.io/projected/221eeffa-0e13-4f2b-b071-954dc973142a-kube-api-access-5l6qt\") pod \"maas-api-revoke-sub-f4dc5e8b-hdfvx\" (UID: \"221eeffa-0e13-4f2b-b071-954dc973142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"
Oct 31 13:29:44.550370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.510970    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:44.550370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.537511    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221eeffa-0e13-4f2b-b071-954dc973142a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4dc5e8b-hdfvx\" (UID: \"221eeffa-0e13-4f2b-b071-954dc973142a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"
Oct 31 13:29:44.601030 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:44.600736    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"
Oct 31 13:29:45.185069 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:45.178855    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:45.191323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:45.190993    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:45.255349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:45.255312    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:45.297296 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:45.297252    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:45.316726 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:45.316675    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:45.354943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:45.317576    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:29:45.354943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:45.319769    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:45.354943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:45.326691    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:45.363233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:45.363102    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:45.374165 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:45.363286    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:29:45.374728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:45.374685    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:29:45.404221 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:45.404104    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:29:45.410411 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:45.410381    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:45.410737 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:45.410679    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:45.537780 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:45.537739    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:45.542831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:45.542787    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:45.563749 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:45.563716    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:29:46.139692 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.139554    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"]
Oct 31 13:29:46.177894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.146730    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"
Oct 31 13:29:46.316897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.292364    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:29:46.332030 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.331996    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:29:46.383893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.347038    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdkcn\" (UniqueName: \"kubernetes.io/projected/cd3967ed-136c-41fe-9f49-e865b6887dd7-kube-api-access-fdkcn\") pod \"maas-api-revoke-sub-967a9954-9tldm\" (UID: \"cd3967ed-136c-41fe-9f49-e865b6887dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"
Oct 31 13:29:46.542421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.522807    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd3967ed-136c-41fe-9f49-e865b6887dd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-967a9954-9tldm\" (UID: \"cd3967ed-136c-41fe-9f49-e865b6887dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"
Oct 31 13:29:46.542421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.350833    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:46.542421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.531825    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:46.548392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.455816    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:29:46.549403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.475684    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:46.549403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.548627    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:46.549403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.501972    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:46.550619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.478432    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:29:46.558371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.455859    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:46.577419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.577384    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:46.579745 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.579712    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:46.655399 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.622903    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:29:46.655399 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.402094    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:46.696612 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.669627    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd3967ed-136c-41fe-9f49-e865b6887dd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-967a9954-9tldm\" (UID: \"cd3967ed-136c-41fe-9f49-e865b6887dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"
Oct 31 13:29:46.954895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.948961    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:29:46.954895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.949010    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"]
Oct 31 13:29:46.954895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.949084    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"]
Oct 31 13:29:46.999600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.999565    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdkcn\" (UniqueName: \"kubernetes.io/projected/cd3967ed-136c-41fe-9f49-e865b6887dd7-kube-api-access-fdkcn\") pod \"maas-api-revoke-sub-967a9954-9tldm\" (UID: \"cd3967ed-136c-41fe-9f49-e865b6887dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"
Oct 31 13:29:46.999849 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.999830    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd3967ed-136c-41fe-9f49-e865b6887dd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-967a9954-9tldm\" (UID: \"cd3967ed-136c-41fe-9f49-e865b6887dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"
Oct 31 13:29:47.000001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:46.999984    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd3967ed-136c-41fe-9f49-e865b6887dd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-967a9954-9tldm\" (UID: \"cd3967ed-136c-41fe-9f49-e865b6887dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"
Oct 31 13:29:47.002302 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.002277    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd3967ed-136c-41fe-9f49-e865b6887dd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-967a9954-9tldm\" (UID: \"cd3967ed-136c-41fe-9f49-e865b6887dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"
Oct 31 13:29:47.007097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.007080    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:29:47.106810 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.106178    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:29:47.142565 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.140508    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:29:47.199954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.153157    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd3967ed-136c-41fe-9f49-e865b6887dd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-967a9954-9tldm\" (UID: \"cd3967ed-136c-41fe-9f49-e865b6887dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"
Oct 31 13:29:47.199954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.156120    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdkcn\" (UniqueName: \"kubernetes.io/projected/cd3967ed-136c-41fe-9f49-e865b6887dd7-kube-api-access-fdkcn\") pod \"maas-api-revoke-sub-967a9954-9tldm\" (UID: \"cd3967ed-136c-41fe-9f49-e865b6887dd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"
Oct 31 13:29:47.199954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.196894    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"
Oct 31 13:29:47.200388 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.200357    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:29:47.200457 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.200417    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:29:47.239207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.239086    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:29:47.249895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.247087    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:47.290895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.283373    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:47.290895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.283426    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:47.387688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.385830    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:47.387688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.385904    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:47.780618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.780583    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx" event={"ID":"221eeffa-0e13-4f2b-b071-954dc973142a","Type":"ContainerStarted","Data":"9e22c386611637a30c14a93f8b41095416afb88dab31b1328046529aeb45f69f"}
Oct 31 13:29:47.780794 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.780648    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx" event={"ID":"221eeffa-0e13-4f2b-b071-954dc973142a","Type":"ContainerStarted","Data":"9589b4399bc48a2f310de8dbd993a9469a0c8c040c187026eacd0c54504e5ffe"}
Oct 31 13:29:47.791040 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:29:47.791010    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3967ed_136c_41fe_9f49_e865b6887dd7.slice/crio-805be904855627315c90a0de12c75ca468512f580519968b8f86878ec5cfb710 WatchSource:0}: Error finding container 805be904855627315c90a0de12c75ca468512f580519968b8f86878ec5cfb710: Status 404 returned error can't find the container with id 805be904855627315c90a0de12c75ca468512f580519968b8f86878ec5cfb710
Oct 31 13:29:47.792553 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.792522    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"]
Oct 31 13:29:47.806340 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:47.806292    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx" podStartSLOduration=4.806276322 podStartE2EDuration="4.806276322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:47.803375831 +0000 UTC m=+1585.201724457" watchObservedRunningTime="2026-10-31 13:29:47.806276322 +0000 UTC m=+1585.204624946"
Oct 31 13:29:48.159432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:48.159393    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:48.254945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:48.253269    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:29:48.254945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:48.253330    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:29:48.369997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:48.369965    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:29:48.370122 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:48.370019    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:29:48.772160 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:29:48.772123    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:29:48.783844 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:48.783817    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:48.788327 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:48.788304    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:29:48.797852 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:48.797825    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm" event={"ID":"cd3967ed-136c-41fe-9f49-e865b6887dd7","Type":"ContainerStarted","Data":"93edfad28f095df4e51d643de483067490c29b8ac6d42ec7d4eba0f6b3a5cddf"}
Oct 31 13:29:48.798100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:48.798080    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm" event={"ID":"cd3967ed-136c-41fe-9f49-e865b6887dd7","Type":"ContainerStarted","Data":"805be904855627315c90a0de12c75ca468512f580519968b8f86878ec5cfb710"}
Oct 31 13:29:48.805415 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:48.805391    4428 generic.go:356] "Generic (PLEG): container finished" podID="221eeffa-0e13-4f2b-b071-954dc973142a" containerID="9e22c386611637a30c14a93f8b41095416afb88dab31b1328046529aeb45f69f" exitCode=0
Oct 31 13:29:48.805818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:48.805798    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx" event={"ID":"221eeffa-0e13-4f2b-b071-954dc973142a","Type":"ContainerDied","Data":"9e22c386611637a30c14a93f8b41095416afb88dab31b1328046529aeb45f69f"}
Oct 31 13:29:48.843889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:48.843340    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm" podStartSLOduration=3.843326907 podStartE2EDuration="3.843326907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:48.837703459 +0000 UTC m=+1586.236052084" watchObservedRunningTime="2026-10-31 13:29:48.843326907 +0000 UTC m=+1586.241675530"
Oct 31 13:29:48.919620 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:48.919586    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:49.728909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:49.722719    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:50.013274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:50.013174    4428 generic.go:356] "Generic (PLEG): container finished" podID="cd3967ed-136c-41fe-9f49-e865b6887dd7" containerID="93edfad28f095df4e51d643de483067490c29b8ac6d42ec7d4eba0f6b3a5cddf" exitCode=0
Oct 31 13:29:50.015902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:50.015855    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm" event={"ID":"cd3967ed-136c-41fe-9f49-e865b6887dd7","Type":"ContainerDied","Data":"93edfad28f095df4e51d643de483067490c29b8ac6d42ec7d4eba0f6b3a5cddf"}
Oct 31 13:29:50.311944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:50.310118    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:50.311944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:50.310182    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:50.438914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:50.433820    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:29:50.438914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:50.433910    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:29:50.630589 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:50.626352    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:50.639347 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:50.638483    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"
Oct 31 13:29:50.917300 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:50.917214    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:50.918804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:50.918776    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:29:50.935065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:50.935046    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/221eeffa-0e13-4f2b-b071-954dc973142a-kube-api-access-5l6qt\" (UniqueName: \"kubernetes.io/projected/221eeffa-0e13-4f2b-b071-954dc973142a-kube-api-access-5l6qt\") pod \"221eeffa-0e13-4f2b-b071-954dc973142a\" (UID: \"221eeffa-0e13-4f2b-b071-954dc973142a\") "
Oct 31 13:29:50.935193 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:50.935079    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/221eeffa-0e13-4f2b-b071-954dc973142a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221eeffa-0e13-4f2b-b071-954dc973142a-maas-api-openshift-ca\") pod \"221eeffa-0e13-4f2b-b071-954dc973142a\" (UID: \"221eeffa-0e13-4f2b-b071-954dc973142a\") "
Oct 31 13:29:50.935193 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:50.935136    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/221eeffa-0e13-4f2b-b071-954dc973142a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221eeffa-0e13-4f2b-b071-954dc973142a-maas-api-xks-ca\") pod \"221eeffa-0e13-4f2b-b071-954dc973142a\" (UID: \"221eeffa-0e13-4f2b-b071-954dc973142a\") "
Oct 31 13:29:50.935458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:50.935433    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221eeffa-0e13-4f2b-b071-954dc973142a-maas-api-openshift-ca" pod "221eeffa-0e13-4f2b-b071-954dc973142a" (UID: "221eeffa-0e13-4f2b-b071-954dc973142a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:29:50.935854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:50.935788    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221eeffa-0e13-4f2b-b071-954dc973142a-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:50.937288 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:50.937265    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221eeffa-0e13-4f2b-b071-954dc973142a-kube-api-access-5l6qt" pod "221eeffa-0e13-4f2b-b071-954dc973142a" (UID: "221eeffa-0e13-4f2b-b071-954dc973142a"). InnerVolumeSpecName "kube-api-access-5l6qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:29:50.937373 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:50.937312    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/221eeffa-0e13-4f2b-b071-954dc973142a-maas-api-xks-ca" pod "221eeffa-0e13-4f2b-b071-954dc973142a" (UID: "221eeffa-0e13-4f2b-b071-954dc973142a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:29:51.034955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.033538    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"
Oct 31 13:29:51.034955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.033558    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx" event={"ID":"221eeffa-0e13-4f2b-b071-954dc973142a","Type":"ContainerDied","Data":"9589b4399bc48a2f310de8dbd993a9469a0c8c040c187026eacd0c54504e5ffe"}
Oct 31 13:29:51.034955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.033612    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9589b4399bc48a2f310de8dbd993a9469a0c8c040c187026eacd0c54504e5ffe"
Oct 31 13:29:51.037961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.037688    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l6qt\" (UniqueName: \"kubernetes.io/projected/221eeffa-0e13-4f2b-b071-954dc973142a-kube-api-access-5l6qt\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:51.037961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.037717    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221eeffa-0e13-4f2b-b071-954dc973142a-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:51.111846 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.110928    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:51.112432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.112411    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:29:51.223330 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.223296    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"
Oct 31 13:29:51.241448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.241423    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd3967ed-136c-41fe-9f49-e865b6887dd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd3967ed-136c-41fe-9f49-e865b6887dd7-maas-api-openshift-ca\") pod \"cd3967ed-136c-41fe-9f49-e865b6887dd7\" (UID: \"cd3967ed-136c-41fe-9f49-e865b6887dd7\") "
Oct 31 13:29:51.241588 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.241544    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd3967ed-136c-41fe-9f49-e865b6887dd7-kube-api-access-fdkcn\" (UniqueName: \"kubernetes.io/projected/cd3967ed-136c-41fe-9f49-e865b6887dd7-kube-api-access-fdkcn\") pod \"cd3967ed-136c-41fe-9f49-e865b6887dd7\" (UID: \"cd3967ed-136c-41fe-9f49-e865b6887dd7\") "
Oct 31 13:29:51.241659 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.241645    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd3967ed-136c-41fe-9f49-e865b6887dd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd3967ed-136c-41fe-9f49-e865b6887dd7-maas-api-xks-ca\") pod \"cd3967ed-136c-41fe-9f49-e865b6887dd7\" (UID: \"cd3967ed-136c-41fe-9f49-e865b6887dd7\") "
Oct 31 13:29:51.241826 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.241795    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd3967ed-136c-41fe-9f49-e865b6887dd7-maas-api-openshift-ca" pod "cd3967ed-136c-41fe-9f49-e865b6887dd7" (UID: "cd3967ed-136c-41fe-9f49-e865b6887dd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:29:51.242193 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.242175    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd3967ed-136c-41fe-9f49-e865b6887dd7-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:51.243607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.243583    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3967ed-136c-41fe-9f49-e865b6887dd7-kube-api-access-fdkcn" pod "cd3967ed-136c-41fe-9f49-e865b6887dd7" (UID: "cd3967ed-136c-41fe-9f49-e865b6887dd7"). InnerVolumeSpecName "kube-api-access-fdkcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:29:51.243810 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.243793    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd3967ed-136c-41fe-9f49-e865b6887dd7-maas-api-xks-ca" pod "cd3967ed-136c-41fe-9f49-e865b6887dd7" (UID: "cd3967ed-136c-41fe-9f49-e865b6887dd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:29:51.343633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.343599    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdkcn\" (UniqueName: \"kubernetes.io/projected/cd3967ed-136c-41fe-9f49-e865b6887dd7-kube-api-access-fdkcn\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:51.343633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.343629    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd3967ed-136c-41fe-9f49-e865b6887dd7-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:51.622892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:51.616414    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:52.042099 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:52.042075    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"
Oct 31 13:29:52.042469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:52.042107    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm" event={"ID":"cd3967ed-136c-41fe-9f49-e865b6887dd7","Type":"ContainerDied","Data":"805be904855627315c90a0de12c75ca468512f580519968b8f86878ec5cfb710"}
Oct 31 13:29:52.042469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:52.042157    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="805be904855627315c90a0de12c75ca468512f580519968b8f86878ec5cfb710"
Oct 31 13:29:52.568902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:52.566692    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:53.446478 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.446434    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:53.802602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.802573    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"]
Oct 31 13:29:53.803229 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.803215    4428 cpu_manager.go:414] "containerMap: removing container" podUID="cd3967ed-136c-41fe-9f49-e865b6887dd7" containerName="revoke-keys"
Oct 31 13:29:53.803294 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.803230    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd3967ed-136c-41fe-9f49-e865b6887dd7" containerName="revoke-keys"
Oct 31 13:29:53.803294 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.803254    4428 cpu_manager.go:414] "containerMap: removing container" podUID="221eeffa-0e13-4f2b-b071-954dc973142a" containerName="revoke-keys"
Oct 31 13:29:53.803294 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.803260    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="221eeffa-0e13-4f2b-b071-954dc973142a" containerName="revoke-keys"
Oct 31 13:29:53.803391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.803360    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd3967ed-136c-41fe-9f49-e865b6887dd7" containerName="revoke-keys"
Oct 31 13:29:53.803391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.803371    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="221eeffa-0e13-4f2b-b071-954dc973142a" containerName="revoke-keys"
Oct 31 13:29:53.807521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.807493    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"
Oct 31 13:29:53.812299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.812271    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:29:53.812577 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.812554    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:29:53.817218 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.817194    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"]
Oct 31 13:29:53.884100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.884070    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cbf5582-v49ql\" (UID: \"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"
Oct 31 13:29:53.884261 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.884138    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9l9c\" (UniqueName: \"kubernetes.io/projected/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-kube-api-access-n9l9c\") pod \"maas-api-revoke-sub-9cbf5582-v49ql\" (UID: \"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"
Oct 31 13:29:53.884261 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.884215    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cbf5582-v49ql\" (UID: \"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"
Oct 31 13:29:53.985251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.985221    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cbf5582-v49ql\" (UID: \"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"
Oct 31 13:29:53.985432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.985279    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9l9c\" (UniqueName: \"kubernetes.io/projected/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-kube-api-access-n9l9c\") pod \"maas-api-revoke-sub-9cbf5582-v49ql\" (UID: \"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"
Oct 31 13:29:53.985432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.985387    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cbf5582-v49ql\" (UID: \"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"
Oct 31 13:29:53.986036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.986016    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cbf5582-v49ql\" (UID: \"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"
Oct 31 13:29:53.987742 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.987708    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cbf5582-v49ql\" (UID: \"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"
Oct 31 13:29:53.997976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:53.997956    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9l9c\" (UniqueName: \"kubernetes.io/projected/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-kube-api-access-n9l9c\") pod \"maas-api-revoke-sub-9cbf5582-v49ql\" (UID: \"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"
Oct 31 13:29:54.130085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:54.130027    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"
Oct 31 13:29:54.403953 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:54.403839    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:54.416641 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:29:54.416611    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2160f0a2_8ac4_46cf_841b_addfbb0e2cf8.slice/crio-5b858001812256ebbda0fb03ff837af5f647f0cc98a9d50852a2e3e2287f8f43 WatchSource:0}: Error finding container 5b858001812256ebbda0fb03ff837af5f647f0cc98a9d50852a2e3e2287f8f43: Status 404 returned error can't find the container with id 5b858001812256ebbda0fb03ff837af5f647f0cc98a9d50852a2e3e2287f8f43
Oct 31 13:29:54.417385 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:54.417324    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"]
Oct 31 13:29:55.067559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:55.067483    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql" event={"ID":"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8","Type":"ContainerStarted","Data":"f1fa4c7b66c91074561341ae7ed6b9d14224005687862750676408e91e65f31a"}
Oct 31 13:29:55.067559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:55.067537    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql" event={"ID":"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8","Type":"ContainerStarted","Data":"5b858001812256ebbda0fb03ff837af5f647f0cc98a9d50852a2e3e2287f8f43"}
Oct 31 13:29:55.091066 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:55.091009    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql" podStartSLOduration=2.090989518 podStartE2EDuration="2.090989518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:55.086903731 +0000 UTC m=+1592.485252357" watchObservedRunningTime="2026-10-31 13:29:55.090989518 +0000 UTC m=+1592.489338143"
Oct 31 13:29:55.296756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:55.282691    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:56.080521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:56.080470    4428 generic.go:356] "Generic (PLEG): container finished" podID="2160f0a2-8ac4-46cf-841b-addfbb0e2cf8" containerID="f1fa4c7b66c91074561341ae7ed6b9d14224005687862750676408e91e65f31a" exitCode=0
Oct 31 13:29:56.081412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:56.081379    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql" event={"ID":"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8","Type":"ContainerDied","Data":"f1fa4c7b66c91074561341ae7ed6b9d14224005687862750676408e91e65f31a"}
Oct 31 13:29:56.142957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:56.142903    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:56.937892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:56.928433    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:57.248525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:57.248499    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"
Oct 31 13:29:57.298402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:57.298368    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-maas-api-xks-ca\") pod \"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8\" (UID: \"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8\") "
Oct 31 13:29:57.298598 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:57.298532    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-kube-api-access-n9l9c\" (UniqueName: \"kubernetes.io/projected/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-kube-api-access-n9l9c\") pod \"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8\" (UID: \"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8\") "
Oct 31 13:29:57.298809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:57.298784    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-maas-api-openshift-ca\") pod \"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8\" (UID: \"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8\") "
Oct 31 13:29:57.299177 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:57.299147    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-maas-api-openshift-ca" pod "2160f0a2-8ac4-46cf-841b-addfbb0e2cf8" (UID: "2160f0a2-8ac4-46cf-841b-addfbb0e2cf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:29:57.299689 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:57.299664    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:57.301505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:57.301478    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-maas-api-xks-ca" pod "2160f0a2-8ac4-46cf-841b-addfbb0e2cf8" (UID: "2160f0a2-8ac4-46cf-841b-addfbb0e2cf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:29:57.302196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:57.302169    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-kube-api-access-n9l9c" pod "2160f0a2-8ac4-46cf-841b-addfbb0e2cf8" (UID: "2160f0a2-8ac4-46cf-841b-addfbb0e2cf8"). InnerVolumeSpecName "kube-api-access-n9l9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:29:57.401174 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:57.401143    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9l9c\" (UniqueName: \"kubernetes.io/projected/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-kube-api-access-n9l9c\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:57.401174 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:57.401174    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:57.877136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:57.870463    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:58.096797 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:58.096769    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"
Oct 31 13:29:58.096977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:58.096807    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql" event={"ID":"2160f0a2-8ac4-46cf-841b-addfbb0e2cf8","Type":"ContainerDied","Data":"5b858001812256ebbda0fb03ff837af5f647f0cc98a9d50852a2e3e2287f8f43"}
Oct 31 13:29:58.096977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:58.096852    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b858001812256ebbda0fb03ff837af5f647f0cc98a9d50852a2e3e2287f8f43"
Oct 31 13:29:58.845892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:58.831311    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:29:58.972173 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:58.972144    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"]
Oct 31 13:29:58.973461 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:58.973425    4428 cpu_manager.go:414] "containerMap: removing container" podUID="2160f0a2-8ac4-46cf-841b-addfbb0e2cf8" containerName="revoke-keys"
Oct 31 13:29:58.973634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:58.973620    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2160f0a2-8ac4-46cf-841b-addfbb0e2cf8" containerName="revoke-keys"
Oct 31 13:29:58.973991 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:58.973975    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="2160f0a2-8ac4-46cf-841b-addfbb0e2cf8" containerName="revoke-keys"
Oct 31 13:29:58.980083 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:58.980065    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"]
Oct 31 13:29:58.980295 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:58.980282    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:58.987758 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:58.987038    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"]
Oct 31 13:29:58.991535 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:58.991496    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:58.993614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:58.993589    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"]
Oct 31 13:29:59.062623 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.062592    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8836a12-5c77-42e2-bb58-e69be67808db-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.062755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.062668    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.062755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.062729    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m2bm\" (UniqueName: \"kubernetes.io/projected/a8836a12-5c77-42e2-bb58-e69be67808db-kube-api-access-7m2bm\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.062835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.062756    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4071546e-73df-4d16-a000-36fc2372f54b-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.062835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.062780    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.062835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.062801    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgclw\" (UniqueName: \"kubernetes.io/projected/4071546e-73df-4d16-a000-36fc2372f54b-kube-api-access-bgclw\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.062985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.062963    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.063039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.063027    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.063144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.063131    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4071546e-73df-4d16-a000-36fc2372f54b-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.063190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.063178    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.063264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.063251    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.063297 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.063275    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4071546e-73df-4d16-a000-36fc2372f54b-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.063333 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.063303    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.063376 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.063365    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.063467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.063454    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.063527 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.063487    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8836a12-5c77-42e2-bb58-e69be67808db-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.063527 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.063508    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8836a12-5c77-42e2-bb58-e69be67808db-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.063649 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.063563    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.164628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.164552    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"]
Oct 31 13:29:59.165073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165041    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4071546e-73df-4d16-a000-36fc2372f54b-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.165188 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165098    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.165188 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165152    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.165321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165200    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.165321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165229    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8836a12-5c77-42e2-bb58-e69be67808db-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.165321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165258    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8836a12-5c77-42e2-bb58-e69be67808db-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.165321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165279    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.165321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165312    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8836a12-5c77-42e2-bb58-e69be67808db-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.165572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165362    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.165572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165385    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m2bm\" (UniqueName: \"kubernetes.io/projected/a8836a12-5c77-42e2-bb58-e69be67808db-kube-api-access-7m2bm\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.165572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165416    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4071546e-73df-4d16-a000-36fc2372f54b-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.165572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165440    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.165572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165459    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgclw\" (UniqueName: \"kubernetes.io/projected/4071546e-73df-4d16-a000-36fc2372f54b-kube-api-access-bgclw\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.165572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165481    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.165572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165534    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.165968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165588    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4071546e-73df-4d16-a000-36fc2372f54b-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.165968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165632    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.165968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.165671    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.166124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.166103    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.166998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.166615    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.166998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.166670    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.166998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.166923    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.167245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.166924    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.167245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.167221    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.167245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.167221    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4071546e-73df-4d16-a000-36fc2372f54b-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.167411 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.167342    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8836a12-5c77-42e2-bb58-e69be67808db-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.167645 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.167610    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.167728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.167652    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.169548 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.169515    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4071546e-73df-4d16-a000-36fc2372f54b-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.169743 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.169722    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.170232 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.170210    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8836a12-5c77-42e2-bb58-e69be67808db-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.170630 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.170612    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.171759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.171730    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.206046 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.206015    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"]
Oct 31 13:29:59.212756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.212702    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"]
Oct 31 13:29:59.213351 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.213321    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8836a12-5c77-42e2-bb58-e69be67808db-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.214759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.214737    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4071546e-73df-4d16-a000-36fc2372f54b-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.216046 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.216015    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m2bm\" (UniqueName: \"kubernetes.io/projected/a8836a12-5c77-42e2-bb58-e69be67808db-kube-api-access-7m2bm\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.216156 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.216074    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgclw\" (UniqueName: \"kubernetes.io/projected/4071546e-73df-4d16-a000-36fc2372f54b-kube-api-access-bgclw\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.231255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.223477    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.256029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.255919    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"]
Oct 31 13:29:59.267394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.267364    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.267579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.267434    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.267724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.267598    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.267806 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.267744    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.268010 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.267988    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.268082 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.268065    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.268227 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.268210    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf6dk\" (UniqueName: \"kubernetes.io/projected/4a514fea-055a-47ce-baa7-cd30cb8a88b4-kube-api-access-rf6dk\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.268368 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.268350    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.268444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.268427    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.298037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.298014    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:29:59.314715 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.314693    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.372686    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.372751    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.372783    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.372834    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.372923    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9bmx\" (UniqueName: \"kubernetes.io/projected/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-kube-api-access-l9bmx\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.372976    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.373028    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.373075    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.373124    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf6dk\" (UniqueName: \"kubernetes.io/projected/4a514fea-055a-47ce-baa7-cd30cb8a88b4-kube-api-access-rf6dk\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.373239    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.373299    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.373326    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.373387    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.373497    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.373522    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.373604    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.373631    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.373708    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.374146    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.374470    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.374735    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.381407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.375561    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.386244 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.386163    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.386351 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.386290    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.388392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.388369    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.394879 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.394823    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.395165 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.395113    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf6dk\" (UniqueName: \"kubernetes.io/projected/4a514fea-055a-47ce-baa7-cd30cb8a88b4-kube-api-access-rf6dk\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.475627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.475598    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.475803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.475633    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.475803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.475717    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.475803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.475768    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.476029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.475851    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.476029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.475908    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.476029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.475937    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.476200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.476088    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.476419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.476388    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.476559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.476486    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.476559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.476523    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9bmx\" (UniqueName: \"kubernetes.io/projected/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-kube-api-access-l9bmx\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.476719 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.476592    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.476770 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.476754    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.476880 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.476831    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.478710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.478691    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.479045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.479025    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.489627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.489583    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.490535 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.490515    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9bmx\" (UniqueName: \"kubernetes.io/projected/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-kube-api-access-l9bmx\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.571680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.571649    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:29:59.581489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.581450    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:29:59.599308 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:29:59.599276    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4071546e_73df_4d16_a000_36fc2372f54b.slice/crio-3df56d956594fb1ca0e8f45a82579e7cb758f5006c8989b32b71ec2f652a3c79 WatchSource:0}: Error finding container 3df56d956594fb1ca0e8f45a82579e7cb758f5006c8989b32b71ec2f652a3c79: Status 404 returned error can't find the container with id 3df56d956594fb1ca0e8f45a82579e7cb758f5006c8989b32b71ec2f652a3c79
Oct 31 13:29:59.600226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.600195    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"]
Oct 31 13:29:59.602436 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.602397    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:59.602526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.602476    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:59.602526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.602506    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:59.646669 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.646114    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:59.646669 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.646211    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:59.646669 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.646257    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:59.651029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.650900    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"]
Oct 31 13:29:59.880089 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:29:59.880049    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a514fea_055a_47ce_baa7_cd30cb8a88b4.slice/crio-dc3a86766f291b4e4f264c0fc27d8ab3d92ee70b82ff9ea25add14f867ee7565 WatchSource:0}: Error finding container dc3a86766f291b4e4f264c0fc27d8ab3d92ee70b82ff9ea25add14f867ee7565: Status 404 returned error can't find the container with id dc3a86766f291b4e4f264c0fc27d8ab3d92ee70b82ff9ea25add14f867ee7565
Oct 31 13:29:59.882796 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.882570    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"]
Oct 31 13:29:59.885374 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.884981    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:59.885374 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.885045    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:59.885374 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.885076    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:59.904099 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:29:59.904068    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbf53aa_dea2_46cb_8bb6_9164ebffb925.slice/crio-580ae4372e90dd75bd383351d8af39485ec0744c8e17e68de27f55f661055f55 WatchSource:0}: Error finding container 580ae4372e90dd75bd383351d8af39485ec0744c8e17e68de27f55f661055f55: Status 404 returned error can't find the container with id 580ae4372e90dd75bd383351d8af39485ec0744c8e17e68de27f55f661055f55
Oct 31 13:29:59.904855 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.904834    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"]
Oct 31 13:29:59.906975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.906947    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:59.907052 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.907007    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:29:59.907052 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:29:59.907043    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:30:00.129801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.120704    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" event={"ID":"ffbf53aa-dea2-46cb-8bb6-9164ebffb925","Type":"ContainerStarted","Data":"35006d3b219885bd7a049ddcd535445877b705b76169fd8d56cbaf360f799119"}
Oct 31 13:30:00.129801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.120765    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" event={"ID":"ffbf53aa-dea2-46cb-8bb6-9164ebffb925","Type":"ContainerStarted","Data":"580ae4372e90dd75bd383351d8af39485ec0744c8e17e68de27f55f661055f55"}
Oct 31 13:30:00.188892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.178711    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:00.188892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.178795    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"]
Oct 31 13:30:00.225775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.215394    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" event={"ID":"4071546e-73df-4d16-a000-36fc2372f54b","Type":"ContainerStarted","Data":"f42e6e218cf025c36b4d6b0ebf897e58ccfe565b42aef9fe335c7cbb8fd7d71a"}
Oct 31 13:30:00.225775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.215446    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" event={"ID":"4071546e-73df-4d16-a000-36fc2372f54b","Type":"ContainerStarted","Data":"3df56d956594fb1ca0e8f45a82579e7cb758f5006c8989b32b71ec2f652a3c79"}
Oct 31 13:30:00.225775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.215594    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"
Oct 31 13:30:00.263529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.252001    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:30:00.263529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.252104    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:30:00.315408 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.305291    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:30:00.316325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.316297    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 13:30:00.316465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.316364    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 13:30:00.346033 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.345997    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrj4t\" (UniqueName: \"kubernetes.io/projected/648da99d-008c-4647-a4c5-0e40d1b932a8-kube-api-access-qrj4t\") pod \"maas-api-revoke-sub-59aafb2f-dj64l\" (UID: \"648da99d-008c-4647-a4c5-0e40d1b932a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"
Oct 31 13:30:00.346414 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.346392    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648da99d-008c-4647-a4c5-0e40d1b932a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59aafb2f-dj64l\" (UID: \"648da99d-008c-4647-a4c5-0e40d1b932a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"
Oct 31 13:30:00.346786 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.346763    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648da99d-008c-4647-a4c5-0e40d1b932a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59aafb2f-dj64l\" (UID: \"648da99d-008c-4647-a4c5-0e40d1b932a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"
Oct 31 13:30:00.349885 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.349837    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" event={"ID":"a8836a12-5c77-42e2-bb58-e69be67808db","Type":"ContainerStarted","Data":"8f11954f00bcb576c130524e86f1593ea416ce35774aff28da5f3613f54b13f7"}
Oct 31 13:30:00.356186 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.356154    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" event={"ID":"a8836a12-5c77-42e2-bb58-e69be67808db","Type":"ContainerStarted","Data":"492cd06048c11ab6dc3a8d2e68baa6c7314578c7ea711ff24714f56cb09ad609"}
Oct 31 13:30:00.357280 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.357240    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" event={"ID":"4a514fea-055a-47ce-baa7-cd30cb8a88b4","Type":"ContainerStarted","Data":"f5919dc9db36dd8b4abdd55cfabb93aa3353b39b8ee3b7812b4bf511fbd213a4"}
Oct 31 13:30:00.357526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.357506    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" event={"ID":"4a514fea-055a-47ce-baa7-cd30cb8a88b4","Type":"ContainerStarted","Data":"dc3a86766f291b4e4f264c0fc27d8ab3d92ee70b82ff9ea25add14f867ee7565"}
Oct 31 13:30:00.367794 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.367773    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9"]
Oct 31 13:30:00.374190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.374170    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9"
Oct 31 13:30:00.378954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.378929    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"]
Oct 31 13:30:00.386913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.384684    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w"]
Oct 31 13:30:00.391460 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.391433    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w"
Oct 31 13:30:00.397818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.397424    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c"]
Oct 31 13:30:00.406663 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.405389    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c"
Oct 31 13:30:00.417486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.414165    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podStartSLOduration=1.414146696 podStartE2EDuration="1.414146696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:00.189889812 +0000 UTC m=+1597.588238437" watchObservedRunningTime="2026-10-31 13:30:00.414146696 +0000 UTC m=+1597.812495321"
Oct 31 13:30:00.417486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.417020    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh"]
Oct 31 13:30:00.430271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.429355    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh"
Oct 31 13:30:00.441469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.441398    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9"]
Oct 31 13:30:00.447959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.447938    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w"]
Oct 31 13:30:00.449836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.449340    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlwbc\" (UniqueName: \"kubernetes.io/projected/84882963-3c7a-411a-9b17-58cdbfd369e0-kube-api-access-dlwbc\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9\" (UID: \"84882963-3c7a-411a-9b17-58cdbfd369e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9"
Oct 31 13:30:00.450323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.450071    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrj4t\" (UniqueName: \"kubernetes.io/projected/648da99d-008c-4647-a4c5-0e40d1b932a8-kube-api-access-qrj4t\") pod \"maas-api-revoke-sub-59aafb2f-dj64l\" (UID: \"648da99d-008c-4647-a4c5-0e40d1b932a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"
Oct 31 13:30:00.462166 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.462135    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648da99d-008c-4647-a4c5-0e40d1b932a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59aafb2f-dj64l\" (UID: \"648da99d-008c-4647-a4c5-0e40d1b932a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"
Oct 31 13:30:00.462943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.462914    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84882963-3c7a-411a-9b17-58cdbfd369e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9\" (UID: \"84882963-3c7a-411a-9b17-58cdbfd369e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9"
Oct 31 13:30:00.467269 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.465630    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648da99d-008c-4647-a4c5-0e40d1b932a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59aafb2f-dj64l\" (UID: \"648da99d-008c-4647-a4c5-0e40d1b932a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"
Oct 31 13:30:00.468247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.468222    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648da99d-008c-4647-a4c5-0e40d1b932a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59aafb2f-dj64l\" (UID: \"648da99d-008c-4647-a4c5-0e40d1b932a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"
Oct 31 13:30:00.473102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.473047    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podStartSLOduration=2.473030618 podStartE2EDuration="2.473030618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:00.252345431 +0000 UTC m=+1597.650694055" watchObservedRunningTime="2026-10-31 13:30:00.473030618 +0000 UTC m=+1597.871379244"
Oct 31 13:30:00.473494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.473270    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh"]
Oct 31 13:30:00.510948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.494400    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648da99d-008c-4647-a4c5-0e40d1b932a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59aafb2f-dj64l\" (UID: \"648da99d-008c-4647-a4c5-0e40d1b932a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"
Oct 31 13:30:00.521942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.521900    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrj4t\" (UniqueName: \"kubernetes.io/projected/648da99d-008c-4647-a4c5-0e40d1b932a8-kube-api-access-qrj4t\") pod \"maas-api-revoke-sub-59aafb2f-dj64l\" (UID: \"648da99d-008c-4647-a4c5-0e40d1b932a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"
Oct 31 13:30:00.587932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.577939    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:30:00.587932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.578442    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c\" (UID: \"3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c"
Oct 31 13:30:00.587932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.578531    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtnw9\" (UniqueName: \"kubernetes.io/projected/f1ff020f-5d77-45ff-b3dc-d89411fc9828-kube-api-access-jtnw9\") pod \"maas-api-key-cleanup-29890890-xz68w\" (UID: \"f1ff020f-5d77-45ff-b3dc-d89411fc9828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w"
Oct 31 13:30:00.587932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.578645    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1512638-7776-4b20-a91f-2be17ade86ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh\" (UID: \"e1512638-7776-4b20-a91f-2be17ade86ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh"
Oct 31 13:30:00.587932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.578775    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlwbc\" (UniqueName: \"kubernetes.io/projected/84882963-3c7a-411a-9b17-58cdbfd369e0-kube-api-access-dlwbc\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9\" (UID: \"84882963-3c7a-411a-9b17-58cdbfd369e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9"
Oct 31 13:30:00.587932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.578856    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpqzn\" (UniqueName: \"kubernetes.io/projected/e1512638-7776-4b20-a91f-2be17ade86ca-kube-api-access-lpqzn\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh\" (UID: \"e1512638-7776-4b20-a91f-2be17ade86ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh"
Oct 31 13:30:00.587932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.578900    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff020f-5d77-45ff-b3dc-d89411fc9828-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890890-xz68w\" (UID: \"f1ff020f-5d77-45ff-b3dc-d89411fc9828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w"
Oct 31 13:30:00.587932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.579010    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84882963-3c7a-411a-9b17-58cdbfd369e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9\" (UID: \"84882963-3c7a-411a-9b17-58cdbfd369e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9"
Oct 31 13:30:00.587932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.579092    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsmbw\" (UniqueName: \"kubernetes.io/projected/3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5-kube-api-access-xsmbw\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c\" (UID: \"3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c"
Oct 31 13:30:00.587932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.579998    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84882963-3c7a-411a-9b17-58cdbfd369e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9\" (UID: \"84882963-3c7a-411a-9b17-58cdbfd369e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9"
Oct 31 13:30:00.587932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.584983    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:30:00.592498 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.592176    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c"]
Oct 31 13:30:00.592639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.592510    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 13:30:00.592639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.592580    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 13:30:00.595983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.595954    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:00.596102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.596007    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:00.597358 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.597342    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlwbc\" (UniqueName: \"kubernetes.io/projected/84882963-3c7a-411a-9b17-58cdbfd369e0-kube-api-access-dlwbc\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9\" (UID: \"84882963-3c7a-411a-9b17-58cdbfd369e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9"
Oct 31 13:30:00.599038 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.598983    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podStartSLOduration=2.598966143 podStartE2EDuration="2.598966143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:00.382299408 +0000 UTC m=+1597.780648046" watchObservedRunningTime="2026-10-31 13:30:00.598966143 +0000 UTC m=+1597.997314768"
Oct 31 13:30:00.607920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.607837    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podStartSLOduration=1.607824795 podStartE2EDuration="1.607824795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:00.411857506 +0000 UTC m=+1597.810206131" watchObservedRunningTime="2026-10-31 13:30:00.607824795 +0000 UTC m=+1598.006173419"
Oct 31 13:30:00.618820 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.618794    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"
Oct 31 13:30:00.688909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.681062    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1512638-7776-4b20-a91f-2be17ade86ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh\" (UID: \"e1512638-7776-4b20-a91f-2be17ade86ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh"
Oct 31 13:30:00.688909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.681250    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpqzn\" (UniqueName: \"kubernetes.io/projected/e1512638-7776-4b20-a91f-2be17ade86ca-kube-api-access-lpqzn\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh\" (UID: \"e1512638-7776-4b20-a91f-2be17ade86ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh"
Oct 31 13:30:00.688909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.681283    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff020f-5d77-45ff-b3dc-d89411fc9828-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890890-xz68w\" (UID: \"f1ff020f-5d77-45ff-b3dc-d89411fc9828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w"
Oct 31 13:30:00.688909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.681426    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsmbw\" (UniqueName: \"kubernetes.io/projected/3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5-kube-api-access-xsmbw\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c\" (UID: \"3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c"
Oct 31 13:30:00.688909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.681497    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c\" (UID: \"3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c"
Oct 31 13:30:00.688909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.681565    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnw9\" (UniqueName: \"kubernetes.io/projected/f1ff020f-5d77-45ff-b3dc-d89411fc9828-kube-api-access-jtnw9\") pod \"maas-api-key-cleanup-29890890-xz68w\" (UID: \"f1ff020f-5d77-45ff-b3dc-d89411fc9828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w"
Oct 31 13:30:00.688909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.682771    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1512638-7776-4b20-a91f-2be17ade86ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh\" (UID: \"e1512638-7776-4b20-a91f-2be17ade86ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh"
Oct 31 13:30:00.690307 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.690249    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c\" (UID: \"3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c"
Oct 31 13:30:00.690479 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.690452    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff020f-5d77-45ff-b3dc-d89411fc9828-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890890-xz68w\" (UID: \"f1ff020f-5d77-45ff-b3dc-d89411fc9828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w"
Oct 31 13:30:00.697136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.697051    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtnw9\" (UniqueName: \"kubernetes.io/projected/f1ff020f-5d77-45ff-b3dc-d89411fc9828-kube-api-access-jtnw9\") pod \"maas-api-key-cleanup-29890890-xz68w\" (UID: \"f1ff020f-5d77-45ff-b3dc-d89411fc9828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w"
Oct 31 13:30:00.698179 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.698152    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpqzn\" (UniqueName: \"kubernetes.io/projected/e1512638-7776-4b20-a91f-2be17ade86ca-kube-api-access-lpqzn\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh\" (UID: \"e1512638-7776-4b20-a91f-2be17ade86ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh"
Oct 31 13:30:00.706539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.706508    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9"
Oct 31 13:30:00.712200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.712153    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsmbw\" (UniqueName: \"kubernetes.io/projected/3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5-kube-api-access-xsmbw\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c\" (UID: \"3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c"
Oct 31 13:30:00.733908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.733829    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w"
Oct 31 13:30:00.760600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.760567    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c"
Oct 31 13:30:00.760761 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:00.760712    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh"
Oct 31 13:30:01.228078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:01.180752    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"]
Oct 31 13:30:01.317415 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:01.316363    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:30:01.328921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:01.318894    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:30:01.328921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:01.318950    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 13:30:01.605046 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:01.593797    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 13:30:01.611033 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:01.605332    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 13:30:01.807930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:01.738196    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:01.937339 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:01.936002    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:01.987898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:01.962512    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:02.126894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.126482    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:02.272839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.238835    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:02.272839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.238916    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:02.272839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.239333    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 13:30:02.272839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.239371    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 13:30:02.452014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.422923    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:30:02.452014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.427949    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 13:30:02.452014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.428172    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 13:30:02.452014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.428206    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 13:30:02.583993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.581273    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:02.583993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.581344    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:02.583993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.582238    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 13:30:02.583993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.582276    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 13:30:02.605096 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.604990    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:02.671828 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.671789    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:02.746512 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.746125    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr" podUID="c20e3218-9e4a-43a1-b46e-050de8eb25c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:02.853781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.853606    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:30:03.057779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:02.976929    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:03.108052 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:03.099647    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:03.197103 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:03.197051    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:03.203575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:03.203535    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:03.206430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:03.206400    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:03.485706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:03.485670    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:30:03.660270 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:03.659402    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:30:03.675397 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:03.675265    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:30:03.781162 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:03.781120    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr" podUID="c20e3218-9e4a-43a1-b46e-050de8eb25c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded"
Oct 31 13:30:03.785899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:03.783414    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Oct 31 13:30:04.038479 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:04.038386    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:04.038954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:04.038925    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:04.629117 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:04.629067    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 13:30:04.650590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:04.650546    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:30:04.735489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:04.735429    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:04.735779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:04.735738    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:05.035962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.035401    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr" podUID="c20e3218-9e4a-43a1-b46e-050de8eb25c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:05.071097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:04.629699    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 13:30:05.071097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.037452    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:05.071097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.038423    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg" podUID="5c86b906-a276-4e67-8e58-79a17118f5fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Oct 31 13:30:05.071097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.039083    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:30:05.071097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:04.889278    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:05.071097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:04.889608    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:05.071097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:04.736201    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 13:30:05.071097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.048655    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 13:30:05.112731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:04.794648    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:30:05.112731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.090313    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 13:30:05.112731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.038170    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:05.112731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.090887    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:05.271959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:04.889734    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:30:05.271959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:04.951238    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6" podUID="149a9b52-ef95-4569-b947-e6e14d707339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:05.271959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:04.951453    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59" podUID="cd0ac048-d186-4f3c-929d-eb870afbc801" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:05.368905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.362071    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 13:30:05.632188 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.632077    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 13:30:05.633426 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.362334    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:30:05.633657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.633624    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 13:30:05.637261 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.402670    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 13:30:05.647024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.646948    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 13:30:05.731910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.677360    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:30:06.234680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.224051    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-4b2ck" podUID="cd510088-f2ae-4abe-9c80-8c7531915aa0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.17:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:06.234680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.952978    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47" podUID="ea978050-20b0-4148-ab41-75844fe1b637" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:06.234680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.953798    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:06.234680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.224752    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:06.234680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.995121    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:30:06.234680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.225073    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 13:30:06.234680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.995189    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 13:30:06.234680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.225131    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 13:30:06.234680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.995583    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 13:30:06.234680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.225177    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 13:30:06.237303 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:05.677989    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:30:06.364932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.334355    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:06.500680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.345603    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59" podUID="cd0ac048-d186-4f3c-929d-eb870afbc801" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:06.501487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.383717    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:06.501676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.501648    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:06.501804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.424640    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:30:06.507908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.424586    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr" podUID="c20e3218-9e4a-43a1-b46e-050de8eb25c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded"
Oct 31 13:30:06.508743 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.424672    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:30:06.590898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.491606    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:30:06.590898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.424692    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:06.590898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.532461    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:30:06.590898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.580835    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 13:30:06.732213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.732161    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:06.773824 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.773768    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 13:30:06.774131 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.774096    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 13:30:06.801657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.801625    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 13:30:06.801926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:06.801893    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 13:30:07.212413 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.150848    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:07.212413 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.151381    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:07.212413 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.151435    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:07.227025 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.226987    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5" podUID="c92cd891-8b41-4511-9fb0-d724cb1caaf5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.51:50052\" within 1s: context deadline exceeded"
Oct 31 13:30:07.704896 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:30:07.227343    4428 logging.go:55] [core] [Channel #886 SubChannel #887]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.51:50052", ServerName: "10.134.0.51:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.51:50052: operation was canceled"
Oct 31 13:30:07.738902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.227402    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:30:07.958966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.414120    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:07.958966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.678539    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:07.958966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.678574    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:30:07.960619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.678594    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:30:07.978391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.969746    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/model-serving-api-75bd89fdfc-6brb8" podUID="874303da-8bf8-44e8-bdb9-d9aadf7dd956" containerName="server" probeResult="failure" output="Get \"https://10.134.0.20:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:07.978391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.706528    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:30:07.978391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.707581    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:30:07.978391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.707612    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:30:08.021903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.678614    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:08.048974 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.742365    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 13:30:08.049322 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.049285    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 13:30:08.049498 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.742424    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:08.049635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.049616    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:08.049771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.744210    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:30:08.055989 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.055945    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 13:30:08.056159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:07.744385    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 13:30:08.056322 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.056290    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 13:30:08.063067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.063037    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr" podUID="c20e3218-9e4a-43a1-b46e-050de8eb25c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded"
Oct 31 13:30:08.139389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.139331    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:08.168623 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.168585    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:08.223085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.214840    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59" podUID="cd0ac048-d186-4f3c-929d-eb870afbc801" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:08.629045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.564450    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 13:30:08.661003 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.660965    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 13:30:08.661215 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.659924    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6" podUID="149a9b52-ef95-4569-b947-e6e14d707339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:08.746904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.594170    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded"
Oct 31 13:30:08.746904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.576683    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:30:08.746904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.743189    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 13:30:08.746904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.659846    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 13:30:08.746904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.743243    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 13:30:08.807911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.807856    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:08.808182 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.808150    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:09.071465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.971120    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:09.071465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.971925    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:09.071465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.974381    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:09.071465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.979378    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:09.071465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.983928    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:09.071465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.984224    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Oct 31 13:30:09.071465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.984274    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:30:09.071465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:08.984508    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:30:09.071465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.031020    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:09.119596 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:09.082962    4428 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="6.322s"
Oct 31 13:30:09.126111 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:30:09.124964    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ff020f_5d77_45ff_b3dc_d89411fc9828.slice/crio-bd254e7700c897e274015a7fe202754a6d9bbaef66da3989dbd2b0d1d89c6a11 WatchSource:0}: Error finding container bd254e7700c897e274015a7fe202754a6d9bbaef66da3989dbd2b0d1d89c6a11: Status 404 returned error can't find the container with id bd254e7700c897e274015a7fe202754a6d9bbaef66da3989dbd2b0d1d89c6a11
Oct 31 13:30:09.174547 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.173977    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:09.328646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.306266    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 13:30:09.328646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.306325    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 13:30:09.328646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.327450    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:30:09.328646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.327509    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 13:30:09.610933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.573573    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:09.686005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.685964    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:09.713040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.624368    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 13:30:09.713258 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.713233    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 13:30:09.739564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.739379    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:09.739564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.739446    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns" podUID="93314798-1f48-408a-b87c-d8f809d2f14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:10.001956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.971372    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:10.044434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.971829    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:10.044850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.044591    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:10.044850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.044642    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl" podUID="7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:10.044850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.987779    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:30:10.045087 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.988349    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:10.045145 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.987728    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:30:10.045774 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.999634    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:10.045774 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.999683    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:30:10.045774 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:09.987760    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:30:10.076234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.070314    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9"]
Oct 31 13:30:10.076234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.070505    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l" event={"ID":"648da99d-008c-4647-a4c5-0e40d1b932a8","Type":"ContainerStarted","Data":"6c10d5bacc8f21d6629da5ea77940e832f3ae222a8046e5c3c23792833d0714a"}
Oct 31 13:30:10.076234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.070541    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c" event={"ID":"3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5","Type":"ContainerStarted","Data":"29f03a7f662cb4322bfd7c8dc4e3498fd1e4dc5da12971aab31cac410b0ad693"}
Oct 31 13:30:10.076234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.070608    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:30:10.076234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.070629    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh" event={"ID":"e1512638-7776-4b20-a91f-2be17ade86ca","Type":"ContainerStarted","Data":"77c358b22fc1a674e184575a09e0d5392b12aae6985875691ac743ce9668188a"}
Oct 31 13:30:10.076234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.070656    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9" event={"ID":"84882963-3c7a-411a-9b17-58cdbfd369e0","Type":"ContainerStarted","Data":"33f37aee9c914e06e4ed6985d36eb75280a0e03e7571d051ccfad1230bc1c618"}
Oct 31 13:30:10.076234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.070684    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh"]
Oct 31 13:30:10.076234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.070710    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c"]
Oct 31 13:30:10.076234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.070739    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"]
Oct 31 13:30:10.111406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.111369    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"]
Oct 31 13:30:10.136938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.111564    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"
Oct 31 13:30:10.137169 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.137045    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:30:10.137983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.137964    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:30:10.138155 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.138141    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:30:10.141979 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.141957    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:30:10.160424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.160400    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:30:10.304716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.270391    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:30:10.304716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.270435    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w"]
Oct 31 13:30:10.304716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.270501    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:30:10.304716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.270541    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:30:10.304716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.270582    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:30:10.304716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.270627    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr"
Oct 31 13:30:10.304716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.270670    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59"
Oct 31 13:30:10.304716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.185162    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:10.376919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.350063    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 13:30:10.376919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.350123    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 13:30:10.377357 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.377323    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:30:10.377512 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.377486    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 13:30:10.701808 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.701523    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:10.722250 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.722196    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:10.729903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.725265    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 13:30:10.729903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.725319    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 13:30:10.760483 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.760441    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6nbx\" (UniqueName: \"kubernetes.io/projected/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-kube-api-access-f6nbx\") pod \"maas-api-revoke-sub-393e027a-fcd8l\" (UID: \"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"
Oct 31 13:30:10.760962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.760937    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-393e027a-fcd8l\" (UID: \"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"
Oct 31 13:30:10.858499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:10.824835    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-393e027a-fcd8l\" (UID: \"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"
Oct 31 13:30:11.083896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.045152    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:30:11.089802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.088191    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:11.431665 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.428293    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6nbx\" (UniqueName: \"kubernetes.io/projected/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-kube-api-access-f6nbx\") pod \"maas-api-revoke-sub-393e027a-fcd8l\" (UID: \"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"
Oct 31 13:30:11.431665 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.428522    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-393e027a-fcd8l\" (UID: \"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"
Oct 31 13:30:11.431665 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.428599    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-393e027a-fcd8l\" (UID: \"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"
Oct 31 13:30:11.438830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.438796    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-393e027a-fcd8l\" (UID: \"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"
Oct 31 13:30:11.474485 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.474450    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 13:30:11.474748 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.474718    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 13:30:11.475010 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.474982    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:30:11.475158 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.475136    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 13:30:11.490687 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.490651    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:11.641005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.493857    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:30:11.779982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.494588    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:30:11.795968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.494712    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:30:11.823237 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.495279    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:11.833948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.576306    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded"
Oct 31 13:30:11.834234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.576389    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:30:11.851019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.591690    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:11.851364 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.851328    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt" podUID="a64a3776-6565-48d0-8c36-5c4d979eeb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:12.212952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.641282    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 13:30:12.212952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.197313    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 13:30:12.212952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.133985    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:12.220509 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:12.019736    4428 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.228s"
Oct 31 13:30:12.220509 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.217721    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l" event={"ID":"648da99d-008c-4647-a4c5-0e40d1b932a8","Type":"ContainerStarted","Data":"c8697012c373505430b03a00d9c0d819f863bbb2acde265c84362ae88d9523f2"}
Oct 31 13:30:12.239353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:11.682056    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:12.239353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.232005    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:12.239353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.158489    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:12.240609 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.240274    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w" event={"ID":"f1ff020f-5d77-45ff-b3dc-d89411fc9828","Type":"ContainerStarted","Data":"bd254e7700c897e274015a7fe202754a6d9bbaef66da3989dbd2b0d1d89c6a11"}
Oct 31 13:30:12.289531 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.289498    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-393e027a-fcd8l\" (UID: \"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"
Oct 31 13:30:12.321896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.314347    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 13:30:12.325896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.322147    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 13:30:12.363895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.340730    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6nbx\" (UniqueName: \"kubernetes.io/projected/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-kube-api-access-f6nbx\") pod \"maas-api-revoke-sub-393e027a-fcd8l\" (UID: \"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"
Oct 31 13:30:12.471457 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.471361    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:12.471704 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.471676    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" podUID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:12.472213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.472186    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:30:12.472388 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.472367    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 13:30:12.640895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.595704    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:12.679945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.679064    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:12.679945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.652625    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:12.789438 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.652581    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:12.818906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.800059    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:12.819288 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.819250    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:12.822893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.819607    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:12.823148 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.823097    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn" podUID="69e04722-598c-4608-9742-440e327e389a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:12.827031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.827007    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:12.827252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.827205    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:12.858421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.831008    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:12.863919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.860484    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:30:12.895769 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.895715    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:12.921631 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.921490    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:12.925225 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:12.925196    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"
Oct 31 13:30:13.071954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.067923    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:13.282067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.213882    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:13.311969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.234327    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:13.352014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.282592    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x" podUID="04337deb-7a21-4e58-8a41-098d8be2e918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:13.602104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.602005    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:13.606002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.605947    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:13.756915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.708060    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr" podUID="c20e3218-9e4a-43a1-b46e-050de8eb25c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": context deadline exceeded"
Oct 31 13:30:13.756915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.710537    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:13.756915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.710588    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:13.756915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.738597    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 13:30:13.756915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.738655    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 13:30:13.756915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.702333    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:13.814198 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.814152    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:13.858643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.833541    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:13.858859 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.851169    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:13.887516 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.887479    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:30:13.889245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.889214    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59" podUID="cd0ac048-d186-4f3c-929d-eb870afbc801" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Oct 31 13:30:13.929436 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.910526    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:13.929991 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.912925    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:13.931649 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.931617    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:13.967053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.967016    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 13:30:13.967321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.967291    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 13:30:13.982712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.982675    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:30:13.982965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:13.982934    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 13:30:14.375920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:14.375856    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:14.536893 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:14.496759    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6231a0_18ab_4d8b_a4ff_0485bf92d2c5.slice/crio-294b2ac362a61c773f2488c94427163a6e7970b588908f045bf2ce36d841652c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84882963_3c7a_411a_9b17_58cdbfd369e0.slice/crio-conmon-32e0381945dd9b719cb3193ad7b728d12e870e8b6e7a98d8fc21daf51b46b61a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84882963_3c7a_411a_9b17_58cdbfd369e0.slice/crio-32e0381945dd9b719cb3193ad7b728d12e870e8b6e7a98d8fc21daf51b46b61a.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 13:30:14.560690 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:14.537544    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:14.925949 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:14.894220    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84882963_3c7a_411a_9b17_58cdbfd369e0.slice/crio-conmon-32e0381945dd9b719cb3193ad7b728d12e870e8b6e7a98d8fc21daf51b46b61a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6231a0_18ab_4d8b_a4ff_0485bf92d2c5.slice/crio-conmon-294b2ac362a61c773f2488c94427163a6e7970b588908f045bf2ce36d841652c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6231a0_18ab_4d8b_a4ff_0485bf92d2c5.slice/crio-294b2ac362a61c773f2488c94427163a6e7970b588908f045bf2ce36d841652c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84882963_3c7a_411a_9b17_58cdbfd369e0.slice/crio-32e0381945dd9b719cb3193ad7b728d12e870e8b6e7a98d8fc21daf51b46b61a.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 13:30:14.995893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:14.926857    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:15.063230 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:14.926908    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:30:15.063230 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:14.965172    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:15.063230 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:14.965210    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:15.063230 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:14.965231    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:30:15.184555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.184452    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:30:15.374505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.374471    4428 generic.go:356] "Generic (PLEG): container finished" podID="84882963-3c7a-411a-9b17-58cdbfd369e0" containerID="32e0381945dd9b719cb3193ad7b728d12e870e8b6e7a98d8fc21daf51b46b61a" exitCode=60
Oct 31 13:30:15.386966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.386929    4428 generic.go:356] "Generic (PLEG): container finished" podID="648da99d-008c-4647-a4c5-0e40d1b932a8" containerID="c8697012c373505430b03a00d9c0d819f863bbb2acde265c84362ae88d9523f2" exitCode=0
Oct 31 13:30:15.552908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.552859    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:30:15.553194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.553151    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 13:30:15.554604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.552982    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 13:30:15.554804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.554780    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 13:30:15.555181 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.555159    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 13:30:15.555335 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.555316    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 13:30:15.555521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.555506    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:30:15.568081 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.568025    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 13:30:15.600213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.600172    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:15.600530 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.600454    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:15.601250 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.601225    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:15.601410 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.601391    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:15.655959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.650240    4428 generic.go:356] "Generic (PLEG): container finished" podID="3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5" containerID="294b2ac362a61c773f2488c94427163a6e7970b588908f045bf2ce36d841652c" exitCode=60
Oct 31 13:30:15.672335 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.656492    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:15.698894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.673209    4428 scope.go:122] "RemoveContainer" containerID="66116041fa913cd7b89e4d5fc59ac1829aa1adc1325461a12d5ce082f52d0a24"
Oct 31 13:30:15.698894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.674515    4428 scope.go:122] "RemoveContainer" containerID="32e0381945dd9b719cb3193ad7b728d12e870e8b6e7a98d8fc21daf51b46b61a"
Oct 31 13:30:15.698894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.675794    4428 scope.go:122] "RemoveContainer" containerID="294b2ac362a61c773f2488c94427163a6e7970b588908f045bf2ce36d841652c"
Oct 31 13:30:15.698894 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:15.680288    4428 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.911s"
Oct 31 13:30:15.698894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.680323    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9" event={"ID":"84882963-3c7a-411a-9b17-58cdbfd369e0","Type":"ContainerDied","Data":"32e0381945dd9b719cb3193ad7b728d12e870e8b6e7a98d8fc21daf51b46b61a"}
Oct 31 13:30:15.698894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.680367    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l" event={"ID":"648da99d-008c-4647-a4c5-0e40d1b932a8","Type":"ContainerDied","Data":"c8697012c373505430b03a00d9c0d819f863bbb2acde265c84362ae88d9523f2"}
Oct 31 13:30:15.812980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.801781    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 13:30:15.812980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.801836    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 13:30:15.812980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.805352    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 13:30:15.812980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:15.805403    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 13:30:16.193808 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:16.142954    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:16.933905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:16.924245    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:17.095284 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:17.095246    4428 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.414s"
Oct 31 13:30:17.095448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.095296    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c" event={"ID":"3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5","Type":"ContainerDied","Data":"294b2ac362a61c773f2488c94427163a6e7970b588908f045bf2ce36d841652c"}
Oct 31 13:30:17.095448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.095364    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:30:17.095448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.095410    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:30:17.095648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.095453    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:30:17.095648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.095490    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:30:17.095648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.095573    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:30:17.095648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.095641    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:30:17.122401 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.110885    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 13:30:17.122625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.122598    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 13:30:17.122958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.112409    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59" podUID="cd0ac048-d186-4f3c-929d-eb870afbc801" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:17.258646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.258613    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:17.291572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.291538    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:30:17.307993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.307935    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 13:30:17.324019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.319355    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:17.324019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.319412    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:17.443407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.443362    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 13:30:17.443772 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.443702    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 13:30:17.517680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.517647    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 13:30:17.521981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.521935    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 13:30:17.560942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.542222    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:30:17.560942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.542287    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:30:17.560942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.542319    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:30:17.560942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.542346    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:30:17.560942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.542379    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:30:17.560942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.542428    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:30:17.560942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.542459    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"]
Oct 31 13:30:17.597403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.596582    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:17.597403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.596655    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:17.597403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.597219    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 13:30:17.597403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.597272    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 13:30:17.658905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.658857    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"
Oct 31 13:30:17.771857 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.770422    4428 generic.go:356] "Generic (PLEG): container finished" podID="e1512638-7776-4b20-a91f-2be17ade86ca" containerID="5c621f7ba916bcb3ff8575c88d4122e202b8ca7d5002a719f67cd96b2aebcd90" exitCode=60
Oct 31 13:30:17.771857 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.770526    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh" event={"ID":"e1512638-7776-4b20-a91f-2be17ade86ca","Type":"ContainerDied","Data":"5c621f7ba916bcb3ff8575c88d4122e202b8ca7d5002a719f67cd96b2aebcd90"}
Oct 31 13:30:17.771857 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.770985    4428 scope.go:122] "RemoveContainer" containerID="5c621f7ba916bcb3ff8575c88d4122e202b8ca7d5002a719f67cd96b2aebcd90"
Oct 31 13:30:17.792958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.790185    4428 generic.go:356] "Generic (PLEG): container finished" podID="f1ff020f-5d77-45ff-b3dc-d89411fc9828" containerID="66116041fa913cd7b89e4d5fc59ac1829aa1adc1325461a12d5ce082f52d0a24" exitCode=60
Oct 31 13:30:17.792958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.790350    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w" event={"ID":"f1ff020f-5d77-45ff-b3dc-d89411fc9828","Type":"ContainerDied","Data":"66116041fa913cd7b89e4d5fc59ac1829aa1adc1325461a12d5ce082f52d0a24"}
Oct 31 13:30:17.815974 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.815948    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/648da99d-008c-4647-a4c5-0e40d1b932a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648da99d-008c-4647-a4c5-0e40d1b932a8-maas-api-openshift-ca\") pod \"648da99d-008c-4647-a4c5-0e40d1b932a8\" (UID: \"648da99d-008c-4647-a4c5-0e40d1b932a8\") "
Oct 31 13:30:17.816670 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.816647    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648da99d-008c-4647-a4c5-0e40d1b932a8-kube-api-access-qrj4t\" (UniqueName: \"kubernetes.io/projected/648da99d-008c-4647-a4c5-0e40d1b932a8-kube-api-access-qrj4t\") pod \"648da99d-008c-4647-a4c5-0e40d1b932a8\" (UID: \"648da99d-008c-4647-a4c5-0e40d1b932a8\") "
Oct 31 13:30:17.817430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.817408    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/648da99d-008c-4647-a4c5-0e40d1b932a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648da99d-008c-4647-a4c5-0e40d1b932a8-maas-api-xks-ca\") pod \"648da99d-008c-4647-a4c5-0e40d1b932a8\" (UID: \"648da99d-008c-4647-a4c5-0e40d1b932a8\") "
Oct 31 13:30:17.818500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.816437    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648da99d-008c-4647-a4c5-0e40d1b932a8-maas-api-openshift-ca" pod "648da99d-008c-4647-a4c5-0e40d1b932a8" (UID: "648da99d-008c-4647-a4c5-0e40d1b932a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:30:17.866204 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.866154    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648da99d-008c-4647-a4c5-0e40d1b932a8-kube-api-access-qrj4t" pod "648da99d-008c-4647-a4c5-0e40d1b932a8" (UID: "648da99d-008c-4647-a4c5-0e40d1b932a8"). InnerVolumeSpecName "kube-api-access-qrj4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:30:17.867782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.867421    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648da99d-008c-4647-a4c5-0e40d1b932a8-maas-api-xks-ca" pod "648da99d-008c-4647-a4c5-0e40d1b932a8" (UID: "648da99d-008c-4647-a4c5-0e40d1b932a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:30:17.868788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.868753    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l" event={"ID":"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0","Type":"ContainerStarted","Data":"f4468823211fe26551845bd68e43fd789b0c2cb1ed91e15ff9e16da0901c9add"}
Oct 31 13:30:17.916942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.916916    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:17.920737 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.920718    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648da99d-008c-4647-a4c5-0e40d1b932a8-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:17.920921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.920906    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648da99d-008c-4647-a4c5-0e40d1b932a8-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:17.921016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:17.921004    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrj4t\" (UniqueName: \"kubernetes.io/projected/648da99d-008c-4647-a4c5-0e40d1b932a8-kube-api-access-qrj4t\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:18.194518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:18.177103    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:30:18.573638 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:18.573256    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:30:18.573638 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:18.573317    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:30:18.786364 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:18.786120    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:30:18.890909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:18.889973    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh" event={"ID":"e1512638-7776-4b20-a91f-2be17ade86ca","Type":"ContainerStarted","Data":"f62e2e22617d329ce4778979a8220db1e3dc1e490706b68fbc2720f4bcb1751f"}
Oct 31 13:30:18.894910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:18.892519    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l" event={"ID":"648da99d-008c-4647-a4c5-0e40d1b932a8","Type":"ContainerDied","Data":"6c10d5bacc8f21d6629da5ea77940e832f3ae222a8046e5c3c23792833d0714a"}
Oct 31 13:30:18.894910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:18.892596    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c10d5bacc8f21d6629da5ea77940e832f3ae222a8046e5c3c23792833d0714a"
Oct 31 13:30:18.894910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:18.892708    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"
Oct 31 13:30:18.963528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:18.944624    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh" podStartSLOduration=18.944603851 podStartE2EDuration="18.944603851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:18.941297076 +0000 UTC m=+1616.339645701" watchObservedRunningTime="2026-10-31 13:30:18.944603851 +0000 UTC m=+1616.342952476"
Oct 31 13:30:18.963528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:18.945157    4428 generic.go:356] "Generic (PLEG): container finished" podID="f1ff020f-5d77-45ff-b3dc-d89411fc9828" containerID="8ab5fc7609828819dc6cc78bbd1c8840c92eab0c928e5d43e37f5e3cdc32d63c" exitCode=60
Oct 31 13:30:18.963528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:18.945330    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w" event={"ID":"f1ff020f-5d77-45ff-b3dc-d89411fc9828","Type":"ContainerDied","Data":"8ab5fc7609828819dc6cc78bbd1c8840c92eab0c928e5d43e37f5e3cdc32d63c"}
Oct 31 13:30:18.963528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:18.945403    4428 scope.go:122] "RemoveContainer" containerID="66116041fa913cd7b89e4d5fc59ac1829aa1adc1325461a12d5ce082f52d0a24"
Oct 31 13:30:18.963528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:18.946314    4428 scope.go:122] "RemoveContainer" containerID="8ab5fc7609828819dc6cc78bbd1c8840c92eab0c928e5d43e37f5e3cdc32d63c"
Oct 31 13:30:18.963528 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:18.946748    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890890-xz68w_odh-ai-gateway-infra(f1ff020f-5d77-45ff-b3dc-d89411fc9828)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w" podUID="f1ff020f-5d77-45ff-b3dc-d89411fc9828"
Oct 31 13:30:18.963528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:18.963422    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:19.019341 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:19.019305    4428 scope.go:122] "RemoveContainer" containerID="2244f74b7fb062869175ff8de0f2fc88fb2db64478b2ef69b72ce2de37e112c2"
Oct 31 13:30:19.020044 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:19.020006    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9_odh-ai-gateway-infra(84882963-3c7a-411a-9b17-58cdbfd369e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9" podUID="84882963-3c7a-411a-9b17-58cdbfd369e0"
Oct 31 13:30:19.021986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:19.021952    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9" event={"ID":"84882963-3c7a-411a-9b17-58cdbfd369e0","Type":"ContainerStarted","Data":"2244f74b7fb062869175ff8de0f2fc88fb2db64478b2ef69b72ce2de37e112c2"}
Oct 31 13:30:19.081084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:19.055905    4428 generic.go:356] "Generic (PLEG): container finished" podID="3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5" containerID="d1259705550a7845ae5d166fe6e78235ee83a79ac9a03b2ee5c42c251557ebe9" exitCode=60
Oct 31 13:30:19.081084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:19.056107    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c" event={"ID":"3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5","Type":"ContainerDied","Data":"d1259705550a7845ae5d166fe6e78235ee83a79ac9a03b2ee5c42c251557ebe9"}
Oct 31 13:30:19.088740 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:19.087101    4428 scope.go:122] "RemoveContainer" containerID="d1259705550a7845ae5d166fe6e78235ee83a79ac9a03b2ee5c42c251557ebe9"
Oct 31 13:30:19.088740 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:19.087709    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c_odh-ai-gateway-infra(3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c" podUID="3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5"
Oct 31 13:30:19.091069 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:19.090962    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l" event={"ID":"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0","Type":"ContainerStarted","Data":"7ac0b0b9a53e2db599852369b48cbd8da05a48290f875b4cd8c68ffba172bc92"}
Oct 31 13:30:19.108017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:19.107934    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:30:19.115101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:19.114045    4428 scope.go:122] "RemoveContainer" containerID="294b2ac362a61c773f2488c94427163a6e7970b588908f045bf2ce36d841652c"
Oct 31 13:30:19.124030 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:19.120146    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:30:19.133913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:19.133222    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:30:19.286096 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:19.286037    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l" podStartSLOduration=15.286016539 podStartE2EDuration="15.286016539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:19.245520909 +0000 UTC m=+1616.643869556" watchObservedRunningTime="2026-10-31 13:30:19.286016539 +0000 UTC m=+1616.684365163"
Oct 31 13:30:19.337802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:19.334388    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:30:19.338545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:19.338282    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 13:30:19.999838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:19.999750    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:20.157215 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:20.157175    4428 generic.go:356] "Generic (PLEG): container finished" podID="84882963-3c7a-411a-9b17-58cdbfd369e0" containerID="2244f74b7fb062869175ff8de0f2fc88fb2db64478b2ef69b72ce2de37e112c2" exitCode=60
Oct 31 13:30:20.157386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:20.157322    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9" event={"ID":"84882963-3c7a-411a-9b17-58cdbfd369e0","Type":"ContainerDied","Data":"2244f74b7fb062869175ff8de0f2fc88fb2db64478b2ef69b72ce2de37e112c2"}
Oct 31 13:30:20.157429 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:20.157383    4428 scope.go:122] "RemoveContainer" containerID="32e0381945dd9b719cb3193ad7b728d12e870e8b6e7a98d8fc21daf51b46b61a"
Oct 31 13:30:20.158002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:20.157984    4428 scope.go:122] "RemoveContainer" containerID="2244f74b7fb062869175ff8de0f2fc88fb2db64478b2ef69b72ce2de37e112c2"
Oct 31 13:30:20.158453 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:20.158426    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9_odh-ai-gateway-infra(84882963-3c7a-411a-9b17-58cdbfd369e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9" podUID="84882963-3c7a-411a-9b17-58cdbfd369e0"
Oct 31 13:30:20.187343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:20.181021    4428 scope.go:122] "RemoveContainer" containerID="d1259705550a7845ae5d166fe6e78235ee83a79ac9a03b2ee5c42c251557ebe9"
Oct 31 13:30:20.198732 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:20.194350    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c_odh-ai-gateway-infra(3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c" podUID="3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5"
Oct 31 13:30:20.198732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:20.197636    4428 generic.go:356] "Generic (PLEG): container finished" podID="e385a07f-5be6-4e14-ba31-3fc5dbbd7de0" containerID="7ac0b0b9a53e2db599852369b48cbd8da05a48290f875b4cd8c68ffba172bc92" exitCode=0
Oct 31 13:30:20.198732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:20.197890    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l" event={"ID":"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0","Type":"ContainerDied","Data":"7ac0b0b9a53e2db599852369b48cbd8da05a48290f875b4cd8c68ffba172bc92"}
Oct 31 13:30:20.204162 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:20.204138    4428 generic.go:356] "Generic (PLEG): container finished" podID="e1512638-7776-4b20-a91f-2be17ade86ca" containerID="f62e2e22617d329ce4778979a8220db1e3dc1e490706b68fbc2720f4bcb1751f" exitCode=60
Oct 31 13:30:20.204404 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:20.204380    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh" event={"ID":"e1512638-7776-4b20-a91f-2be17ade86ca","Type":"ContainerDied","Data":"f62e2e22617d329ce4778979a8220db1e3dc1e490706b68fbc2720f4bcb1751f"}
Oct 31 13:30:20.204601 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:20.204589    4428 scope.go:122] "RemoveContainer" containerID="f62e2e22617d329ce4778979a8220db1e3dc1e490706b68fbc2720f4bcb1751f"
Oct 31 13:30:20.204964 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:20.204946    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh_odh-ai-gateway-infra(e1512638-7776-4b20-a91f-2be17ade86ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh" podUID="e1512638-7776-4b20-a91f-2be17ade86ca"
Oct 31 13:30:20.211469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:20.209519    4428 scope.go:122] "RemoveContainer" containerID="8ab5fc7609828819dc6cc78bbd1c8840c92eab0c928e5d43e37f5e3cdc32d63c"
Oct 31 13:30:20.211469 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:20.209882    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890890-xz68w_odh-ai-gateway-infra(f1ff020f-5d77-45ff-b3dc-d89411fc9828)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w" podUID="f1ff020f-5d77-45ff-b3dc-d89411fc9828"
Oct 31 13:30:20.233744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:20.233719    4428 scope.go:122] "RemoveContainer" containerID="5c621f7ba916bcb3ff8575c88d4122e202b8ca7d5002a719f67cd96b2aebcd90"
Oct 31 13:30:20.614115 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:20.614007    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:20.614115 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:20.614076    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:21.018914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:21.018858    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:21.194085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:21.194054    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:30:21.207131 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:21.207099    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:30:21.217732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:21.217709    4428 scope.go:122] "RemoveContainer" containerID="f62e2e22617d329ce4778979a8220db1e3dc1e490706b68fbc2720f4bcb1751f"
Oct 31 13:30:21.218226 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:21.218193    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh_odh-ai-gateway-infra(e1512638-7776-4b20-a91f-2be17ade86ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh" podUID="e1512638-7776-4b20-a91f-2be17ade86ca"
Oct 31 13:30:21.219467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:21.219443    4428 scope.go:122] "RemoveContainer" containerID="2244f74b7fb062869175ff8de0f2fc88fb2db64478b2ef69b72ce2de37e112c2"
Oct 31 13:30:21.220070 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:21.219790    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9_odh-ai-gateway-infra(84882963-3c7a-411a-9b17-58cdbfd369e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9" podUID="84882963-3c7a-411a-9b17-58cdbfd369e0"
Oct 31 13:30:21.437317 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:21.437290    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"
Oct 31 13:30:21.543208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:21.543126    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-kube-api-access-f6nbx\" (UniqueName: \"kubernetes.io/projected/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-kube-api-access-f6nbx\") pod \"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0\" (UID: \"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0\") "
Oct 31 13:30:21.543363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:21.543301    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-maas-api-openshift-ca\") pod \"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0\" (UID: \"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0\") "
Oct 31 13:30:21.543363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:21.543343    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-maas-api-xks-ca\") pod \"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0\" (UID: \"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0\") "
Oct 31 13:30:21.543648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:21.543612    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-maas-api-openshift-ca" pod "e385a07f-5be6-4e14-ba31-3fc5dbbd7de0" (UID: "e385a07f-5be6-4e14-ba31-3fc5dbbd7de0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:30:21.544051 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:21.544025    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:21.545474 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:21.545451    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-kube-api-access-f6nbx" pod "e385a07f-5be6-4e14-ba31-3fc5dbbd7de0" (UID: "e385a07f-5be6-4e14-ba31-3fc5dbbd7de0"). InnerVolumeSpecName "kube-api-access-f6nbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:30:21.545568 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:21.545514    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-maas-api-xks-ca" pod "e385a07f-5be6-4e14-ba31-3fc5dbbd7de0" (UID: "e385a07f-5be6-4e14-ba31-3fc5dbbd7de0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:30:21.645877 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:21.645838    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6nbx\" (UniqueName: \"kubernetes.io/projected/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-kube-api-access-f6nbx\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:21.646205 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:21.645909    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:22.173888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:22.172991    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:22.246712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:22.246620    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l" event={"ID":"e385a07f-5be6-4e14-ba31-3fc5dbbd7de0","Type":"ContainerDied","Data":"f4468823211fe26551845bd68e43fd789b0c2cb1ed91e15ff9e16da0901c9add"}
Oct 31 13:30:22.246712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:22.246682    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4468823211fe26551845bd68e43fd789b0c2cb1ed91e15ff9e16da0901c9add"
Oct 31 13:30:22.246989 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:22.246760    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"
Oct 31 13:30:23.026894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:23.022756    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:23.903954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:23.903804    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:24.282290 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:24.282254    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:24.522306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:24.522257    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:24.977824 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:24.977763    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:25.150511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:25.150300    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:26.104916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:26.104569    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:27.320897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:27.317969    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:28.323906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:28.320325    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:28.747947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:28.747916    4428 scope.go:122] "RemoveContainer" containerID="8ab5fc7609828819dc6cc78bbd1c8840c92eab0c928e5d43e37f5e3cdc32d63c"
Oct 31 13:30:28.789931 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:28.789893    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:29.243814 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:29.243774    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:29.420594 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:29.420482    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:29.496439 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:29.496006    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w" event={"ID":"f1ff020f-5d77-45ff-b3dc-d89411fc9828","Type":"ContainerStarted","Data":"f3e8b6043a086be7de1027f428693de6ff1f7400cb418ca6879a78a9ff8cf176"}
Oct 31 13:30:29.571916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:29.571218    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w" podStartSLOduration=29.571193094 podStartE2EDuration="29.571193094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:29.553681607 +0000 UTC m=+1626.952030232" watchObservedRunningTime="2026-10-31 13:30:29.571193094 +0000 UTC m=+1626.969541732"
Oct 31 13:30:30.390941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:30.389333    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:30.512409 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:30.512370    4428 generic.go:356] "Generic (PLEG): container finished" podID="f1ff020f-5d77-45ff-b3dc-d89411fc9828" containerID="f3e8b6043a086be7de1027f428693de6ff1f7400cb418ca6879a78a9ff8cf176" exitCode=60
Oct 31 13:30:30.512842 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:30.512560    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w" event={"ID":"f1ff020f-5d77-45ff-b3dc-d89411fc9828","Type":"ContainerDied","Data":"f3e8b6043a086be7de1027f428693de6ff1f7400cb418ca6879a78a9ff8cf176"}
Oct 31 13:30:30.512842 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:30.512652    4428 scope.go:122] "RemoveContainer" containerID="8ab5fc7609828819dc6cc78bbd1c8840c92eab0c928e5d43e37f5e3cdc32d63c"
Oct 31 13:30:30.513217 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:30.513187    4428 scope.go:122] "RemoveContainer" containerID="f3e8b6043a086be7de1027f428693de6ff1f7400cb418ca6879a78a9ff8cf176"
Oct 31 13:30:30.513590 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:30.513570    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890890-xz68w_odh-ai-gateway-infra(f1ff020f-5d77-45ff-b3dc-d89411fc9828)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w" podUID="f1ff020f-5d77-45ff-b3dc-d89411fc9828"
Oct 31 13:30:30.560044 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:30.560005    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w"]
Oct 31 13:30:30.748066 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:30.748038    4428 scope.go:122] "RemoveContainer" containerID="2244f74b7fb062869175ff8de0f2fc88fb2db64478b2ef69b72ce2de37e112c2"
Oct 31 13:30:31.429891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:31.428502    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:31.558451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:31.557471    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9" event={"ID":"84882963-3c7a-411a-9b17-58cdbfd369e0","Type":"ContainerStarted","Data":"72867c2eca528f70a605a5e071dc2cf3d3a67e2362fd44db0b417d1cc10ebba9"}
Oct 31 13:30:31.747302 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:31.747233    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9" podStartSLOduration=31.747209977 podStartE2EDuration="31.747209977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:31.624304165 +0000 UTC m=+1629.022652789" watchObservedRunningTime="2026-10-31 13:30:31.747209977 +0000 UTC m=+1629.145558603"
Oct 31 13:30:31.758922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:31.758290    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:31.863573 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:31.863547    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w"
Oct 31 13:30:31.920830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:31.920804    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ff020f-5d77-45ff-b3dc-d89411fc9828-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff020f-5d77-45ff-b3dc-d89411fc9828-maas-api-service-ca\") pod \"f1ff020f-5d77-45ff-b3dc-d89411fc9828\" (UID: \"f1ff020f-5d77-45ff-b3dc-d89411fc9828\") "
Oct 31 13:30:31.921081 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:31.921064    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ff020f-5d77-45ff-b3dc-d89411fc9828-kube-api-access-jtnw9\" (UniqueName: \"kubernetes.io/projected/f1ff020f-5d77-45ff-b3dc-d89411fc9828-kube-api-access-jtnw9\") pod \"f1ff020f-5d77-45ff-b3dc-d89411fc9828\" (UID: \"f1ff020f-5d77-45ff-b3dc-d89411fc9828\") "
Oct 31 13:30:31.921276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:31.921242    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ff020f-5d77-45ff-b3dc-d89411fc9828-maas-api-service-ca" pod "f1ff020f-5d77-45ff-b3dc-d89411fc9828" (UID: "f1ff020f-5d77-45ff-b3dc-d89411fc9828"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:30:31.921778 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:31.921754    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff020f-5d77-45ff-b3dc-d89411fc9828-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:31.923453 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:31.923426    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ff020f-5d77-45ff-b3dc-d89411fc9828-kube-api-access-jtnw9" pod "f1ff020f-5d77-45ff-b3dc-d89411fc9828" (UID: "f1ff020f-5d77-45ff-b3dc-d89411fc9828"). InnerVolumeSpecName "kube-api-access-jtnw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:30:31.944657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:31.944632    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:32.023329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:32.023298    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtnw9\" (UniqueName: \"kubernetes.io/projected/f1ff020f-5d77-45ff-b3dc-d89411fc9828-kube-api-access-jtnw9\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:32.566725 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:32.566700    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w"
Oct 31 13:30:32.567200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:32.566733    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w" event={"ID":"f1ff020f-5d77-45ff-b3dc-d89411fc9828","Type":"ContainerDied","Data":"bd254e7700c897e274015a7fe202754a6d9bbaef66da3989dbd2b0d1d89c6a11"}
Oct 31 13:30:32.567200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:32.566799    4428 scope.go:122] "RemoveContainer" containerID="f3e8b6043a086be7de1027f428693de6ff1f7400cb418ca6879a78a9ff8cf176"
Oct 31 13:30:32.576701 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:32.576673    4428 generic.go:356] "Generic (PLEG): container finished" podID="84882963-3c7a-411a-9b17-58cdbfd369e0" containerID="72867c2eca528f70a605a5e071dc2cf3d3a67e2362fd44db0b417d1cc10ebba9" exitCode=60
Oct 31 13:30:32.576939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:32.576908    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9" event={"ID":"84882963-3c7a-411a-9b17-58cdbfd369e0","Type":"ContainerDied","Data":"72867c2eca528f70a605a5e071dc2cf3d3a67e2362fd44db0b417d1cc10ebba9"}
Oct 31 13:30:32.577658 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:32.577437    4428 scope.go:122] "RemoveContainer" containerID="72867c2eca528f70a605a5e071dc2cf3d3a67e2362fd44db0b417d1cc10ebba9"
Oct 31 13:30:32.577847 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:32.577821    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9_odh-ai-gateway-infra(84882963-3c7a-411a-9b17-58cdbfd369e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9" podUID="84882963-3c7a-411a-9b17-58cdbfd369e0"
Oct 31 13:30:32.583387 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:32.583360    4428 scope.go:122] "RemoveContainer" containerID="2244f74b7fb062869175ff8de0f2fc88fb2db64478b2ef69b72ce2de37e112c2"
Oct 31 13:30:32.621402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:32.621369    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:32.640350 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:32.640321    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w"]
Oct 31 13:30:32.648688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:32.648130    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9"]
Oct 31 13:30:32.655138 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:32.655115    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890890-xz68w"]
Oct 31 13:30:32.753147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:32.753118    4428 scope.go:122] "RemoveContainer" containerID="d1259705550a7845ae5d166fe6e78235ee83a79ac9a03b2ee5c42c251557ebe9"
Oct 31 13:30:32.758783 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:32.758762    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ff020f-5d77-45ff-b3dc-d89411fc9828" path="/var/lib/kubelet/pods/f1ff020f-5d77-45ff-b3dc-d89411fc9828/volumes"
Oct 31 13:30:32.808119 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:32.808090    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:32.978233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:32.977478    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:33.590638 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:33.590602    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c" event={"ID":"3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5","Type":"ContainerStarted","Data":"2828f85ebbe9152b011d3e5793d98cf951a824c6a78e1fa0f676c4dff0375525"}
Oct 31 13:30:33.631316 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:33.631278    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:33.641424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:33.641368    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c" podStartSLOduration=33.64135161 podStartE2EDuration="33.64135161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:33.639640604 +0000 UTC m=+1631.037989221" watchObservedRunningTime="2026-10-31 13:30:33.64135161 +0000 UTC m=+1631.039700241"
Oct 31 13:30:33.748447 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:33.748422    4428 scope.go:122] "RemoveContainer" containerID="f62e2e22617d329ce4778979a8220db1e3dc1e490706b68fbc2720f4bcb1751f"
Oct 31 13:30:33.809909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:33.809889    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9"
Oct 31 13:30:33.819952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:33.819925    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:33.875274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:33.875187    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84882963-3c7a-411a-9b17-58cdbfd369e0-kube-api-access-dlwbc\" (UniqueName: \"kubernetes.io/projected/84882963-3c7a-411a-9b17-58cdbfd369e0-kube-api-access-dlwbc\") pod \"84882963-3c7a-411a-9b17-58cdbfd369e0\" (UID: \"84882963-3c7a-411a-9b17-58cdbfd369e0\") "
Oct 31 13:30:33.875757 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:33.875286    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84882963-3c7a-411a-9b17-58cdbfd369e0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84882963-3c7a-411a-9b17-58cdbfd369e0-maas-api-service-ca\") pod \"84882963-3c7a-411a-9b17-58cdbfd369e0\" (UID: \"84882963-3c7a-411a-9b17-58cdbfd369e0\") "
Oct 31 13:30:33.876135 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:33.876105    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84882963-3c7a-411a-9b17-58cdbfd369e0-maas-api-service-ca" pod "84882963-3c7a-411a-9b17-58cdbfd369e0" (UID: "84882963-3c7a-411a-9b17-58cdbfd369e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:30:33.878322 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:33.878299    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84882963-3c7a-411a-9b17-58cdbfd369e0-kube-api-access-dlwbc" pod "84882963-3c7a-411a-9b17-58cdbfd369e0" (UID: "84882963-3c7a-411a-9b17-58cdbfd369e0"). InnerVolumeSpecName "kube-api-access-dlwbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:30:33.977153 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:33.977121    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84882963-3c7a-411a-9b17-58cdbfd369e0-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:33.977153 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:33.977147    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlwbc\" (UniqueName: \"kubernetes.io/projected/84882963-3c7a-411a-9b17-58cdbfd369e0-kube-api-access-dlwbc\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:34.002804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.002776    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:34.036804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.036776    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"]
Oct 31 13:30:34.037521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037504    4428 cpu_manager.go:414] "containerMap: removing container" podUID="648da99d-008c-4647-a4c5-0e40d1b932a8" containerName="revoke-keys"
Oct 31 13:30:34.037581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037523    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648da99d-008c-4647-a4c5-0e40d1b932a8" containerName="revoke-keys"
Oct 31 13:30:34.037581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037532    4428 cpu_manager.go:414] "containerMap: removing container" podUID="84882963-3c7a-411a-9b17-58cdbfd369e0" containerName="cleanup"
Oct 31 13:30:34.037581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037537    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84882963-3c7a-411a-9b17-58cdbfd369e0" containerName="cleanup"
Oct 31 13:30:34.037581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037556    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e385a07f-5be6-4e14-ba31-3fc5dbbd7de0" containerName="revoke-keys"
Oct 31 13:30:34.037581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037561    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e385a07f-5be6-4e14-ba31-3fc5dbbd7de0" containerName="revoke-keys"
Oct 31 13:30:34.037581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037572    4428 cpu_manager.go:414] "containerMap: removing container" podUID="f1ff020f-5d77-45ff-b3dc-d89411fc9828" containerName="cleanup"
Oct 31 13:30:34.037581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037576    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ff020f-5d77-45ff-b3dc-d89411fc9828" containerName="cleanup"
Oct 31 13:30:34.037581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037581    4428 cpu_manager.go:414] "containerMap: removing container" podUID="84882963-3c7a-411a-9b17-58cdbfd369e0" containerName="cleanup"
Oct 31 13:30:34.037890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037586    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84882963-3c7a-411a-9b17-58cdbfd369e0" containerName="cleanup"
Oct 31 13:30:34.037890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037622    4428 cpu_manager.go:414] "containerMap: removing container" podUID="f1ff020f-5d77-45ff-b3dc-d89411fc9828" containerName="cleanup"
Oct 31 13:30:34.037890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037626    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ff020f-5d77-45ff-b3dc-d89411fc9828" containerName="cleanup"
Oct 31 13:30:34.037890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037713    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="84882963-3c7a-411a-9b17-58cdbfd369e0" containerName="cleanup"
Oct 31 13:30:34.037890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037724    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ff020f-5d77-45ff-b3dc-d89411fc9828" containerName="cleanup"
Oct 31 13:30:34.037890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037732    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ff020f-5d77-45ff-b3dc-d89411fc9828" containerName="cleanup"
Oct 31 13:30:34.037890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037737    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="648da99d-008c-4647-a4c5-0e40d1b932a8" containerName="revoke-keys"
Oct 31 13:30:34.037890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037745    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="84882963-3c7a-411a-9b17-58cdbfd369e0" containerName="cleanup"
Oct 31 13:30:34.037890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.037752    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e385a07f-5be6-4e14-ba31-3fc5dbbd7de0" containerName="revoke-keys"
Oct 31 13:30:34.042986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.042965    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"
Oct 31 13:30:34.056414 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.056356    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c7b990ba\""
Oct 31 13:30:34.062828 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.062799    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"]
Oct 31 13:30:34.078439 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.078406    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e49afd3-26bd-493f-a807-ccd12c325d74-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv\" (UID: \"7e49afd3-26bd-493f-a807-ccd12c325d74\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"
Oct 31 13:30:34.078651 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.078630    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jfs4\" (UniqueName: \"kubernetes.io/projected/7e49afd3-26bd-493f-a807-ccd12c325d74-kube-api-access-7jfs4\") pod \"payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv\" (UID: \"7e49afd3-26bd-493f-a807-ccd12c325d74\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"
Oct 31 13:30:34.082202 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.082177    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"]
Oct 31 13:30:34.083300 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.083278    4428 cpu_manager.go:414] "containerMap: removing container" podUID="f1ff020f-5d77-45ff-b3dc-d89411fc9828" containerName="cleanup"
Oct 31 13:30:34.083300 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.083300    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ff020f-5d77-45ff-b3dc-d89411fc9828" containerName="cleanup"
Oct 31 13:30:34.083465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.083318    4428 cpu_manager.go:414] "containerMap: removing container" podUID="84882963-3c7a-411a-9b17-58cdbfd369e0" containerName="cleanup"
Oct 31 13:30:34.083465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.083325    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84882963-3c7a-411a-9b17-58cdbfd369e0" containerName="cleanup"
Oct 31 13:30:34.083656 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.083640    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ff020f-5d77-45ff-b3dc-d89411fc9828" containerName="cleanup"
Oct 31 13:30:34.083720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.083673    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="84882963-3c7a-411a-9b17-58cdbfd369e0" containerName="cleanup"
Oct 31 13:30:34.089255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.089223    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"
Oct 31 13:30:34.097748 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.097725    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c7b990ba-dockercfg-466bk\""
Oct 31 13:30:34.099198 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.099172    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"]
Oct 31 13:30:34.193484 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.193314    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:34.225295 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.218599    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e49afd3-26bd-493f-a807-ccd12c325d74-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv\" (UID: \"7e49afd3-26bd-493f-a807-ccd12c325d74\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"
Oct 31 13:30:34.270894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.269986    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2tl2\" (UniqueName: \"kubernetes.io/projected/a5ef0238-0e04-42d5-93b2-625f05dbaeb2-kube-api-access-l2tl2\") pod \"payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829\" (UID: \"a5ef0238-0e04-42d5-93b2-625f05dbaeb2\") " pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"
Oct 31 13:30:34.270894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.270279    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jfs4\" (UniqueName: \"kubernetes.io/projected/7e49afd3-26bd-493f-a807-ccd12c325d74-kube-api-access-7jfs4\") pod \"payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv\" (UID: \"7e49afd3-26bd-493f-a807-ccd12c325d74\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"
Oct 31 13:30:34.270894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.270403    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ef0238-0e04-42d5-93b2-625f05dbaeb2-plugins-config-volume\") pod \"payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829\" (UID: \"a5ef0238-0e04-42d5-93b2-625f05dbaeb2\") " pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"
Oct 31 13:30:34.270894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.227806    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e49afd3-26bd-493f-a807-ccd12c325d74-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv\" (UID: \"7e49afd3-26bd-493f-a807-ccd12c325d74\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"
Oct 31 13:30:34.349896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.341771    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jfs4\" (UniqueName: \"kubernetes.io/projected/7e49afd3-26bd-493f-a807-ccd12c325d74-kube-api-access-7jfs4\") pod \"payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv\" (UID: \"7e49afd3-26bd-493f-a807-ccd12c325d74\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"
Oct 31 13:30:34.378688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.377198    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ef0238-0e04-42d5-93b2-625f05dbaeb2-plugins-config-volume\") pod \"payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829\" (UID: \"a5ef0238-0e04-42d5-93b2-625f05dbaeb2\") " pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"
Oct 31 13:30:34.378688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.377369    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2tl2\" (UniqueName: \"kubernetes.io/projected/a5ef0238-0e04-42d5-93b2-625f05dbaeb2-kube-api-access-l2tl2\") pod \"payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829\" (UID: \"a5ef0238-0e04-42d5-93b2-625f05dbaeb2\") " pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"
Oct 31 13:30:34.378688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.378630    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ef0238-0e04-42d5-93b2-625f05dbaeb2-plugins-config-volume\") pod \"payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829\" (UID: \"a5ef0238-0e04-42d5-93b2-625f05dbaeb2\") " pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"
Oct 31 13:30:34.388166 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.388134    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"
Oct 31 13:30:34.399842 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.399814    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2tl2\" (UniqueName: \"kubernetes.io/projected/a5ef0238-0e04-42d5-93b2-625f05dbaeb2-kube-api-access-l2tl2\") pod \"payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829\" (UID: \"a5ef0238-0e04-42d5-93b2-625f05dbaeb2\") " pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"
Oct 31 13:30:34.455546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.455513    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"
Oct 31 13:30:34.728077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.727446    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c"]
Oct 31 13:30:34.756947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.750977    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:34.959018 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:34.958976    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:35.047224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.045189    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9"
Oct 31 13:30:35.050702 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.050312    4428 generic.go:356] "Generic (PLEG): container finished" podID="3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5" containerID="2828f85ebbe9152b011d3e5793d98cf951a824c6a78e1fa0f676c4dff0375525" exitCode=60
Oct 31 13:30:35.120966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.086039    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh" event={"ID":"e1512638-7776-4b20-a91f-2be17ade86ca","Type":"ContainerStarted","Data":"bcd3896b0c38484e9153179855cb176d08838e77c74b64ed2f6b3d206a01fc28"}
Oct 31 13:30:35.120966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.086113    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9" event={"ID":"84882963-3c7a-411a-9b17-58cdbfd369e0","Type":"ContainerDied","Data":"33f37aee9c914e06e4ed6985d36eb75280a0e03e7571d051ccfad1230bc1c618"}
Oct 31 13:30:35.120966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.086139    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c" event={"ID":"3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5","Type":"ContainerDied","Data":"2828f85ebbe9152b011d3e5793d98cf951a824c6a78e1fa0f676c4dff0375525"}
Oct 31 13:30:35.120966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.086178    4428 scope.go:122] "RemoveContainer" containerID="72867c2eca528f70a605a5e071dc2cf3d3a67e2362fd44db0b417d1cc10ebba9"
Oct 31 13:30:35.223904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.222930    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9"]
Oct 31 13:30:35.275906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.275613    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890890-xbrn9"]
Oct 31 13:30:35.323464 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.323382    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:35.364041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.362903    4428 scope.go:122] "RemoveContainer" containerID="d1259705550a7845ae5d166fe6e78235ee83a79ac9a03b2ee5c42c251557ebe9"
Oct 31 13:30:35.659891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.649465    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"]
Oct 31 13:30:35.686595 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:30:35.686568    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ef0238_0e04_42d5_93b2_625f05dbaeb2.slice/crio-6a42bb62c6a05ef65e2d1ea6ebe5732f7b0233e12b08bdc4b69c3d5cd58d3a2f WatchSource:0}: Error finding container 6a42bb62c6a05ef65e2d1ea6ebe5732f7b0233e12b08bdc4b69c3d5cd58d3a2f: Status 404 returned error can't find the container with id 6a42bb62c6a05ef65e2d1ea6ebe5732f7b0233e12b08bdc4b69c3d5cd58d3a2f
Oct 31 13:30:35.715242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.715217    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c"
Oct 31 13:30:35.768083 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.767323    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"]
Oct 31 13:30:35.843817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.843791    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5-kube-api-access-xsmbw\" (UniqueName: \"kubernetes.io/projected/3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5-kube-api-access-xsmbw\") pod \"3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5\" (UID: \"3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5\") "
Oct 31 13:30:35.843978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.843940    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5-maas-api-service-ca\") pod \"3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5\" (UID: \"3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5\") "
Oct 31 13:30:35.844356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.844328    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5-maas-api-service-ca" pod "3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5" (UID: "3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:30:35.844699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.844680    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:35.845997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.845975    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5-kube-api-access-xsmbw" pod "3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5" (UID: "3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5"). InnerVolumeSpecName "kube-api-access-xsmbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:30:35.946282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:35.946252    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsmbw\" (UniqueName: \"kubernetes.io/projected/3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5-kube-api-access-xsmbw\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:36.062700 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.062602    4428 generic.go:356] "Generic (PLEG): container finished" podID="e1512638-7776-4b20-a91f-2be17ade86ca" containerID="bcd3896b0c38484e9153179855cb176d08838e77c74b64ed2f6b3d206a01fc28" exitCode=60
Oct 31 13:30:36.062903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.062699    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh" event={"ID":"e1512638-7776-4b20-a91f-2be17ade86ca","Type":"ContainerDied","Data":"bcd3896b0c38484e9153179855cb176d08838e77c74b64ed2f6b3d206a01fc28"}
Oct 31 13:30:36.062903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.062810    4428 scope.go:122] "RemoveContainer" containerID="f62e2e22617d329ce4778979a8220db1e3dc1e490706b68fbc2720f4bcb1751f"
Oct 31 13:30:36.063274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.063238    4428 scope.go:122] "RemoveContainer" containerID="bcd3896b0c38484e9153179855cb176d08838e77c74b64ed2f6b3d206a01fc28"
Oct 31 13:30:36.063651 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:36.063622    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh_odh-ai-gateway-infra(e1512638-7776-4b20-a91f-2be17ade86ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh" podUID="e1512638-7776-4b20-a91f-2be17ade86ca"
Oct 31 13:30:36.066757 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.066735    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829" event={"ID":"a5ef0238-0e04-42d5-93b2-625f05dbaeb2","Type":"ContainerStarted","Data":"a7bf51823cc388bb4ea437997fe510787c89469904dc39820b98d3baaaf45aac"}
Oct 31 13:30:36.066836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.066775    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829" event={"ID":"a5ef0238-0e04-42d5-93b2-625f05dbaeb2","Type":"ContainerStarted","Data":"6a42bb62c6a05ef65e2d1ea6ebe5732f7b0233e12b08bdc4b69c3d5cd58d3a2f"}
Oct 31 13:30:36.066836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.066803    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"
Oct 31 13:30:36.068385 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.068368    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c"
Oct 31 13:30:36.068470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.068401    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c" event={"ID":"3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5","Type":"ContainerDied","Data":"29f03a7f662cb4322bfd7c8dc4e3498fd1e4dc5da12971aab31cac410b0ad693"}
Oct 31 13:30:36.070042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.070020    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv" event={"ID":"7e49afd3-26bd-493f-a807-ccd12c325d74","Type":"ContainerStarted","Data":"472047b68ae71da26ca3363fbcae4644c270047302a782e1ad1632d86f1588e6"}
Oct 31 13:30:36.070137 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.070065    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"
Oct 31 13:30:36.070137 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.070087    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv" event={"ID":"7e49afd3-26bd-493f-a807-ccd12c325d74","Type":"ContainerStarted","Data":"79ad9df6592b6c272865196e87057c33e24c372df6d8fef4f9386f1766998cfa"}
Oct 31 13:30:36.082352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.082328    4428 scope.go:122] "RemoveContainer" containerID="2828f85ebbe9152b011d3e5793d98cf951a824c6a78e1fa0f676c4dff0375525"
Oct 31 13:30:36.129227 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.129176    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv" podStartSLOduration=2.129160405 podStartE2EDuration="2.129160405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:36.127008921 +0000 UTC m=+1633.525357570" watchObservedRunningTime="2026-10-31 13:30:36.129160405 +0000 UTC m=+1633.527509030"
Oct 31 13:30:36.167739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.167691    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829" podStartSLOduration=2.167676247 podStartE2EDuration="2.167676247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:36.164218597 +0000 UTC m=+1633.562567221" watchObservedRunningTime="2026-10-31 13:30:36.167676247 +0000 UTC m=+1633.566024870"
Oct 31 13:30:36.186410 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.186377    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c"]
Oct 31 13:30:36.191894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.191850    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890890-gxs7c"]
Oct 31 13:30:36.195901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.195859    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh"]
Oct 31 13:30:36.357100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.347958    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:36.758003 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.757974    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5" path="/var/lib/kubelet/pods/3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5/volumes"
Oct 31 13:30:36.758514 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:36.758498    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84882963-3c7a-411a-9b17-58cdbfd369e0" path="/var/lib/kubelet/pods/84882963-3c7a-411a-9b17-58cdbfd369e0/volumes"
Oct 31 13:30:37.170431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.170387    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"]
Oct 31 13:30:37.171563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.171536    4428 cpu_manager.go:414] "containerMap: removing container" podUID="3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5" containerName="cleanup"
Oct 31 13:30:37.171563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.171563    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5" containerName="cleanup"
Oct 31 13:30:37.171746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.171590    4428 cpu_manager.go:414] "containerMap: removing container" podUID="3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5" containerName="cleanup"
Oct 31 13:30:37.171746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.171598    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5" containerName="cleanup"
Oct 31 13:30:37.171746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.171619    4428 cpu_manager.go:414] "containerMap: removing container" podUID="3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5" containerName="cleanup"
Oct 31 13:30:37.171746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.171628    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5" containerName="cleanup"
Oct 31 13:30:37.172067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.171841    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5" containerName="cleanup"
Oct 31 13:30:37.172067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.171859    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5" containerName="cleanup"
Oct 31 13:30:37.172067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.171887    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d6231a0-18ab-4d8b-a4ff-0485bf92d2c5" containerName="cleanup"
Oct 31 13:30:37.188916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.181680    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"
Oct 31 13:30:37.188916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.186504    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:30:37.188916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.186614    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"]
Oct 31 13:30:37.243132 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.243103    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:37.266410 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.266379    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh"
Oct 31 13:30:37.303998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.303961    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f726de20-47fe-4518-b6b9-cdd3a3d579e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2123aec9-vnc9h\" (UID: \"f726de20-47fe-4518-b6b9-cdd3a3d579e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"
Oct 31 13:30:37.304244 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.304224    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f726de20-47fe-4518-b6b9-cdd3a3d579e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2123aec9-vnc9h\" (UID: \"f726de20-47fe-4518-b6b9-cdd3a3d579e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"
Oct 31 13:30:37.304383 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.304360    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nmgb\" (UniqueName: \"kubernetes.io/projected/f726de20-47fe-4518-b6b9-cdd3a3d579e3-kube-api-access-8nmgb\") pod \"maas-api-revoke-sub-2123aec9-vnc9h\" (UID: \"f726de20-47fe-4518-b6b9-cdd3a3d579e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"
Oct 31 13:30:37.406947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.406602    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1512638-7776-4b20-a91f-2be17ade86ca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1512638-7776-4b20-a91f-2be17ade86ca-maas-api-service-ca\") pod \"e1512638-7776-4b20-a91f-2be17ade86ca\" (UID: \"e1512638-7776-4b20-a91f-2be17ade86ca\") "
Oct 31 13:30:37.406947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.406838    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1512638-7776-4b20-a91f-2be17ade86ca-kube-api-access-lpqzn\" (UniqueName: \"kubernetes.io/projected/e1512638-7776-4b20-a91f-2be17ade86ca-kube-api-access-lpqzn\") pod \"e1512638-7776-4b20-a91f-2be17ade86ca\" (UID: \"e1512638-7776-4b20-a91f-2be17ade86ca\") "
Oct 31 13:30:37.407344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.407324    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f726de20-47fe-4518-b6b9-cdd3a3d579e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2123aec9-vnc9h\" (UID: \"f726de20-47fe-4518-b6b9-cdd3a3d579e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"
Oct 31 13:30:37.408892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.407471    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nmgb\" (UniqueName: \"kubernetes.io/projected/f726de20-47fe-4518-b6b9-cdd3a3d579e3-kube-api-access-8nmgb\") pod \"maas-api-revoke-sub-2123aec9-vnc9h\" (UID: \"f726de20-47fe-4518-b6b9-cdd3a3d579e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"
Oct 31 13:30:37.408892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.407599    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f726de20-47fe-4518-b6b9-cdd3a3d579e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2123aec9-vnc9h\" (UID: \"f726de20-47fe-4518-b6b9-cdd3a3d579e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"
Oct 31 13:30:37.408892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.408572    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f726de20-47fe-4518-b6b9-cdd3a3d579e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2123aec9-vnc9h\" (UID: \"f726de20-47fe-4518-b6b9-cdd3a3d579e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"
Oct 31 13:30:37.422895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.409887    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1512638-7776-4b20-a91f-2be17ade86ca-maas-api-service-ca" pod "e1512638-7776-4b20-a91f-2be17ade86ca" (UID: "e1512638-7776-4b20-a91f-2be17ade86ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:30:37.422895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.413901    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:37.476936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.473417    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f726de20-47fe-4518-b6b9-cdd3a3d579e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2123aec9-vnc9h\" (UID: \"f726de20-47fe-4518-b6b9-cdd3a3d579e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"
Oct 31 13:30:37.476936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.474623    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1512638-7776-4b20-a91f-2be17ade86ca-kube-api-access-lpqzn" pod "e1512638-7776-4b20-a91f-2be17ade86ca" (UID: "e1512638-7776-4b20-a91f-2be17ade86ca"). InnerVolumeSpecName "kube-api-access-lpqzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:30:37.500994 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.498959    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nmgb\" (UniqueName: \"kubernetes.io/projected/f726de20-47fe-4518-b6b9-cdd3a3d579e3-kube-api-access-8nmgb\") pod \"maas-api-revoke-sub-2123aec9-vnc9h\" (UID: \"f726de20-47fe-4518-b6b9-cdd3a3d579e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"
Oct 31 13:30:37.521893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.519192    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpqzn\" (UniqueName: \"kubernetes.io/projected/e1512638-7776-4b20-a91f-2be17ade86ca-kube-api-access-lpqzn\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:37.521893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.519231    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1512638-7776-4b20-a91f-2be17ade86ca-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:37.530956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.528236    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"
Oct 31 13:30:37.618895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.617658    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:37.995972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:37.995933    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:38.043776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:38.043749    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"]
Oct 31 13:30:38.044212 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:30:38.044186    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf726de20_47fe_4518_b6b9_cdd3a3d579e3.slice/crio-e76f21d8dcb8611f3609bb1aed41031e3a42d33dfea282cf8f6df1aac3ef2b05 WatchSource:0}: Error finding container e76f21d8dcb8611f3609bb1aed41031e3a42d33dfea282cf8f6df1aac3ef2b05: Status 404 returned error can't find the container with id e76f21d8dcb8611f3609bb1aed41031e3a42d33dfea282cf8f6df1aac3ef2b05
Oct 31 13:30:38.108887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:38.106788    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h" event={"ID":"f726de20-47fe-4518-b6b9-cdd3a3d579e3","Type":"ContainerStarted","Data":"e76f21d8dcb8611f3609bb1aed41031e3a42d33dfea282cf8f6df1aac3ef2b05"}
Oct 31 13:30:38.111039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:38.109764    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh"
Oct 31 13:30:38.111039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:38.109768    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh" event={"ID":"e1512638-7776-4b20-a91f-2be17ade86ca","Type":"ContainerDied","Data":"77c358b22fc1a674e184575a09e0d5392b12aae6985875691ac743ce9668188a"}
Oct 31 13:30:38.111039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:38.109832    4428 scope.go:122] "RemoveContainer" containerID="bcd3896b0c38484e9153179855cb176d08838e77c74b64ed2f6b3d206a01fc28"
Oct 31 13:30:38.208025 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:38.196742    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:38.281251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:38.279843    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh"]
Oct 31 13:30:38.462896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:38.461433    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:38.570942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:38.570829    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890890-4x5kh"]
Oct 31 13:30:39.105890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.096482    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1512638-7776-4b20-a91f-2be17ade86ca" path="/var/lib/kubelet/pods/e1512638-7776-4b20-a91f-2be17ade86ca/volumes"
Oct 31 13:30:39.160689 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.160659    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"]
Oct 31 13:30:39.162152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.162125    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e1512638-7776-4b20-a91f-2be17ade86ca" containerName="cleanup"
Oct 31 13:30:39.162310 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.162293    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1512638-7776-4b20-a91f-2be17ade86ca" containerName="cleanup"
Oct 31 13:30:39.162417 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.162407    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e1512638-7776-4b20-a91f-2be17ade86ca" containerName="cleanup"
Oct 31 13:30:39.162490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.162481    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1512638-7776-4b20-a91f-2be17ade86ca" containerName="cleanup"
Oct 31 13:30:39.162621 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.162612    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e1512638-7776-4b20-a91f-2be17ade86ca" containerName="cleanup"
Oct 31 13:30:39.162688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.162679    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1512638-7776-4b20-a91f-2be17ade86ca" containerName="cleanup"
Oct 31 13:30:39.162966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.162950    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1512638-7776-4b20-a91f-2be17ade86ca" containerName="cleanup"
Oct 31 13:30:39.163069 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.163060    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1512638-7776-4b20-a91f-2be17ade86ca" containerName="cleanup"
Oct 31 13:30:39.163139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.163131    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1512638-7776-4b20-a91f-2be17ade86ca" containerName="cleanup"
Oct 31 13:30:39.249187 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.249153    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"]
Oct 31 13:30:39.251976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.251848    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"
Oct 31 13:30:39.325893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.323749    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67f6fbe5-txghh\" (UID: \"54553f5b-58e8-45da-a20c-bdd60e1dcfd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"
Oct 31 13:30:39.325893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.323820    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr5db\" (UniqueName: \"kubernetes.io/projected/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-kube-api-access-rr5db\") pod \"maas-api-revoke-sub-67f6fbe5-txghh\" (UID: \"54553f5b-58e8-45da-a20c-bdd60e1dcfd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"
Oct 31 13:30:39.325893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.324225    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67f6fbe5-txghh\" (UID: \"54553f5b-58e8-45da-a20c-bdd60e1dcfd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"
Oct 31 13:30:39.434070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.432353    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67f6fbe5-txghh\" (UID: \"54553f5b-58e8-45da-a20c-bdd60e1dcfd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"
Oct 31 13:30:39.434070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.432452    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67f6fbe5-txghh\" (UID: \"54553f5b-58e8-45da-a20c-bdd60e1dcfd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"
Oct 31 13:30:39.434070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.432501    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr5db\" (UniqueName: \"kubernetes.io/projected/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-kube-api-access-rr5db\") pod \"maas-api-revoke-sub-67f6fbe5-txghh\" (UID: \"54553f5b-58e8-45da-a20c-bdd60e1dcfd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"
Oct 31 13:30:39.435179 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.435151    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67f6fbe5-txghh\" (UID: \"54553f5b-58e8-45da-a20c-bdd60e1dcfd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"
Oct 31 13:30:39.450213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.449903    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67f6fbe5-txghh\" (UID: \"54553f5b-58e8-45da-a20c-bdd60e1dcfd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"
Oct 31 13:30:39.462748 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.462677    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr5db\" (UniqueName: \"kubernetes.io/projected/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-kube-api-access-rr5db\") pod \"maas-api-revoke-sub-67f6fbe5-txghh\" (UID: \"54553f5b-58e8-45da-a20c-bdd60e1dcfd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"
Oct 31 13:30:39.662452 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.662419    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"
Oct 31 13:30:39.687742 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.687691    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:39.833168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.832994    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:39.833912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.833859    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:30:39.947256 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.946069    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:39.947256 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.946778    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:30:39.971102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.963894    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:39.971102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:39.964187    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:40.121976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:40.119132    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:40.260897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:40.256972    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:40.397959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:40.383455    4428 generic.go:356] "Generic (PLEG): container finished" podID="f726de20-47fe-4518-b6b9-cdd3a3d579e3" containerID="f6b6296850d7f9a92093a7f85f5422992e8f94455a15aa04d7d75db10aa1d422" exitCode=0
Oct 31 13:30:40.397959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:40.383662    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h" event={"ID":"f726de20-47fe-4518-b6b9-cdd3a3d579e3","Type":"ContainerDied","Data":"f6b6296850d7f9a92093a7f85f5422992e8f94455a15aa04d7d75db10aa1d422"}
Oct 31 13:30:40.560898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:40.552689    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:40.694959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:40.690967    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:40.694959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:40.691444    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:30:40.945644 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:40.945532    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:40.966906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:40.963652    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:40.970924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:40.968955    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:40.970924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:40.969371    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:30:40.974928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:40.972306    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:30:41.000721 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:41.000690    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:30:41.026096 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:41.001777    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:41.045910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:41.030968    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:30:41.045910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:41.031035    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:30:41.045910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:41.031073    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:30:41.045910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:41.031103    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:30:41.128016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:41.127972    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:41.477176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:41.425650    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:30:41.524574 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:41.440605    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:30:41.815265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:41.782081    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:30:41.999933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:41.951326    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:41.999933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:41.972161    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:41.999933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:41.994148    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:42.411208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:42.340548    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:42.503046 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:42.485830    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47" podUID="ea978050-20b0-4148-ab41-75844fe1b637" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:42.612945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:42.506745    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:42.639770 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:42.637842    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:30:42.969993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:42.939092    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:30:43.102926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:43.060595    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:43.102926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:43.070573    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:43.102926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:43.087297    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:30:43.102926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:43.087481    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:30:43.177897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:43.152003    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:43.177897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:43.152106    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:43.424595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:43.333554    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:43.424595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:43.333623    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" podUID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:43.424595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:43.387949    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:43.578973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:43.451940    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:30:43.852199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:43.478052    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59" podUID="cd0ac048-d186-4f3c-929d-eb870afbc801" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Oct 31 13:30:43.852199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:43.499048    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:43.852199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:43.825404    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:43.852199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:43.795602    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8" podUID="bf16863d-965a-458c-bf0d-966d50096325" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 31 13:30:43.852199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:43.827946    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:30:44.041599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:43.931545    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:44.041599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:43.940338    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:44.052100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:44.051996    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:44.085885 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:44.084307    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:44.087294 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:44.087263    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:44.087562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:44.087538    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:30:44.087760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:44.087738    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:44.498914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:44.494004    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:44.653894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:44.651656    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:44.738896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:44.731591    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh" event={"ID":"54553f5b-58e8-45da-a20c-bdd60e1dcfd6","Type":"ContainerStarted","Data":"4bc55e0962c3197ca5afb50cfac47a1a65a62093c4a5862e53f202dae08029e2"}
Oct 31 13:30:44.886521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:44.816804    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:30:44.886521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:44.836468    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:44.949146 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:44.923572    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:45.002955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:44.950196    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:30:45.203459 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.139950    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:45.203459 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.140383    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:45.223241 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.223182    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:30:45.224004 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.223960    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:30:45.290624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.290534    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"]
Oct 31 13:30:45.334114 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.334082    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"]
Oct 31 13:30:45.335370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.334393    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"
Oct 31 13:30:45.390930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.390900    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8439b78a\""
Oct 31 13:30:45.417906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.408454    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8439b78a-dockercfg-q6dt7\""
Oct 31 13:30:45.498386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.485325    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4b37cc-b081-4b6b-a71b-21b6cfc5766f-plugins-config-volume\") pod \"payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs\" (UID: \"8e4b37cc-b081-4b6b-a71b-21b6cfc5766f\") " pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"
Oct 31 13:30:45.498386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.485698    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxn8s\" (UniqueName: \"kubernetes.io/projected/8e4b37cc-b081-4b6b-a71b-21b6cfc5766f-kube-api-access-hxn8s\") pod \"payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs\" (UID: \"8e4b37cc-b081-4b6b-a71b-21b6cfc5766f\") " pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"
Oct 31 13:30:45.537223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.511741    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:45.620168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.609126    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxn8s\" (UniqueName: \"kubernetes.io/projected/8e4b37cc-b081-4b6b-a71b-21b6cfc5766f-kube-api-access-hxn8s\") pod \"payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs\" (UID: \"8e4b37cc-b081-4b6b-a71b-21b6cfc5766f\") " pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"
Oct 31 13:30:45.624124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.624081    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4b37cc-b081-4b6b-a71b-21b6cfc5766f-plugins-config-volume\") pod \"payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs\" (UID: \"8e4b37cc-b081-4b6b-a71b-21b6cfc5766f\") " pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"
Oct 31 13:30:45.637243 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.616133    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:45.697941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.675390    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4b37cc-b081-4b6b-a71b-21b6cfc5766f-plugins-config-volume\") pod \"payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs\" (UID: \"8e4b37cc-b081-4b6b-a71b-21b6cfc5766f\") " pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"
Oct 31 13:30:45.831194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.776463    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"]
Oct 31 13:30:45.831194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.781543    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxn8s\" (UniqueName: \"kubernetes.io/projected/8e4b37cc-b081-4b6b-a71b-21b6cfc5766f-kube-api-access-hxn8s\") pod \"payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs\" (UID: \"8e4b37cc-b081-4b6b-a71b-21b6cfc5766f\") " pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"
Oct 31 13:30:45.909015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.880857    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:30:45.915386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.915346    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:46.008072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.912565    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"
Oct 31 13:30:46.008072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:45.966451    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:46.239352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:46.147575    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:46.239352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:46.151840    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh" event={"ID":"54553f5b-58e8-45da-a20c-bdd60e1dcfd6","Type":"ContainerStarted","Data":"4b40ef8be59b885389d176d71a073a3820c936c78f2523d505e6c5335e480854"}
Oct 31 13:30:46.239352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:46.191243    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:30:46.239352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:46.229429    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:46.239352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:46.229828    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:30:46.657969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:46.656319    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:46.950279 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:46.950166    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:47.231858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:47.231812    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:47.249956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:47.072433    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:47.376534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:47.371059    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:47.477925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:47.439906    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:30:47.477925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:47.440225    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:47.477925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:47.454622    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:30:48.017962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:47.894108    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:48.186900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:47.903006    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded"
Oct 31 13:30:48.264895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:48.261122    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh" podStartSLOduration=10.261099838 podStartE2EDuration="10.261099838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:47.246296964 +0000 UTC m=+1644.644645602" watchObservedRunningTime="2026-10-31 13:30:48.261099838 +0000 UTC m=+1645.659448462"
Oct 31 13:30:48.300985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:48.264133    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:48.311253 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:48.204496    4428 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.449s"
Oct 31 13:30:48.311819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:48.311792    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"
Oct 31 13:30:48.312121 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:48.312106    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"
Oct 31 13:30:48.312600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:48.285470    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:30:48.315611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:48.315579    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:30:48.352353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:48.341532    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:48.394895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:48.375833    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:30:48.438124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:48.438049    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:48.438497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:48.438431    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:48.441713 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:48.441682    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:48.467747 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:48.467676    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:48.484568 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:48.484523    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:48.485563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:48.485528    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:48.785718 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:48.783678    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:48.847833 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:48.847795    4428 generic.go:356] "Generic (PLEG): container finished" podID="54553f5b-58e8-45da-a20c-bdd60e1dcfd6" containerID="4b40ef8be59b885389d176d71a073a3820c936c78f2523d505e6c5335e480854" exitCode=0
Oct 31 13:30:49.191955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.187315    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:49.191955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.189972    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"
Oct 31 13:30:49.262467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.261158    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:49.278976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.263493    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh" event={"ID":"54553f5b-58e8-45da-a20c-bdd60e1dcfd6","Type":"ContainerDied","Data":"4b40ef8be59b885389d176d71a073a3820c936c78f2523d505e6c5335e480854"}
Oct 31 13:30:49.306049 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.305959    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"]
Oct 31 13:30:49.324895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.307358    4428 cpu_manager.go:414] "containerMap: removing container" podUID="f726de20-47fe-4518-b6b9-cdd3a3d579e3" containerName="revoke-keys"
Oct 31 13:30:49.325148 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.325120    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f726de20-47fe-4518-b6b9-cdd3a3d579e3" containerName="revoke-keys"
Oct 31 13:30:49.325708 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.325683    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f726de20-47fe-4518-b6b9-cdd3a3d579e3" containerName="revoke-keys"
Oct 31 13:30:49.427917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.280244    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f726de20-47fe-4518-b6b9-cdd3a3d579e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f726de20-47fe-4518-b6b9-cdd3a3d579e3-maas-api-openshift-ca\") pod \"f726de20-47fe-4518-b6b9-cdd3a3d579e3\" (UID: \"f726de20-47fe-4518-b6b9-cdd3a3d579e3\") "
Oct 31 13:30:49.427917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.371068    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f726de20-47fe-4518-b6b9-cdd3a3d579e3-kube-api-access-8nmgb\" (UniqueName: \"kubernetes.io/projected/f726de20-47fe-4518-b6b9-cdd3a3d579e3-kube-api-access-8nmgb\") pod \"f726de20-47fe-4518-b6b9-cdd3a3d579e3\" (UID: \"f726de20-47fe-4518-b6b9-cdd3a3d579e3\") "
Oct 31 13:30:49.427917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.371353    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f726de20-47fe-4518-b6b9-cdd3a3d579e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f726de20-47fe-4518-b6b9-cdd3a3d579e3-maas-api-xks-ca\") pod \"f726de20-47fe-4518-b6b9-cdd3a3d579e3\" (UID: \"f726de20-47fe-4518-b6b9-cdd3a3d579e3\") "
Oct 31 13:30:49.427917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.376619    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:49.427917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.396951    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:49.427917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.397615    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:30:49.444437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.444375    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:49.448231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.445091    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:49.491289 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.491247    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:49.506452 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.493901    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:49.556960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.508405    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:49.666527 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.557497    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f726de20-47fe-4518-b6b9-cdd3a3d579e3-maas-api-openshift-ca" pod "f726de20-47fe-4518-b6b9-cdd3a3d579e3" (UID: "f726de20-47fe-4518-b6b9-cdd3a3d579e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:30:49.719897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.714102    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"
Oct 31 13:30:49.752893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.742521    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f726de20-47fe-4518-b6b9-cdd3a3d579e3-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:49.799894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.755160    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fc1bbe2b\""
Oct 31 13:30:49.799894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.755971    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fc1bbe2b-dockercfg-bg49w\""
Oct 31 13:30:49.824077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.823990    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"]
Oct 31 13:30:49.888926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:49.888885    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f726de20-47fe-4518-b6b9-cdd3a3d579e3-maas-api-xks-ca" pod "f726de20-47fe-4518-b6b9-cdd3a3d579e3" (UID: "f726de20-47fe-4518-b6b9-cdd3a3d579e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:30:50.017891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.013735    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f726de20-47fe-4518-b6b9-cdd3a3d579e3-kube-api-access-8nmgb" pod "f726de20-47fe-4518-b6b9-cdd3a3d579e3" (UID: "f726de20-47fe-4518-b6b9-cdd3a3d579e3"). InnerVolumeSpecName "kube-api-access-8nmgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:30:50.064897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.023940    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nllv\" (UniqueName: \"kubernetes.io/projected/8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa-kube-api-access-5nllv\") pod \"payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p\" (UID: \"8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"
Oct 31 13:30:50.064897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.024115    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p\" (UID: \"8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"
Oct 31 13:30:50.064897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.024343    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f726de20-47fe-4518-b6b9-cdd3a3d579e3-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:50.064897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.024363    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nmgb\" (UniqueName: \"kubernetes.io/projected/f726de20-47fe-4518-b6b9-cdd3a3d579e3-kube-api-access-8nmgb\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:50.104431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.076038    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Oct 31 13:30:50.189904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.154664    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nllv\" (UniqueName: \"kubernetes.io/projected/8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa-kube-api-access-5nllv\") pod \"payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p\" (UID: \"8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"
Oct 31 13:30:50.219437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.219403    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p\" (UID: \"8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"
Oct 31 13:30:50.380402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.336016    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:50.380402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.360999    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:50.424552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.424512    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:50.424951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.424915    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:50.425238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.425217    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:50.425376 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.425355    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:50.628490 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:50.628448    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:30:50.677932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.650708    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"
Oct 31 13:30:51.041916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.758779    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nllv\" (UniqueName: \"kubernetes.io/projected/8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa-kube-api-access-5nllv\") pod \"payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p\" (UID: \"8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"
Oct 31 13:30:51.050547 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.809859    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:51.075437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.810202    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h" event={"ID":"f726de20-47fe-4518-b6b9-cdd3a3d579e3","Type":"ContainerDied","Data":"e76f21d8dcb8611f3609bb1aed41031e3a42d33dfea282cf8f6df1aac3ef2b05"}
Oct 31 13:30:51.095303 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:51.095265    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e76f21d8dcb8611f3609bb1aed41031e3a42d33dfea282cf8f6df1aac3ef2b05"
Oct 31 13:30:51.191952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:51.125424    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:51.191952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:51.175835    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded"
Oct 31 13:30:51.191952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.811069    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:51.191952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.811681    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:51.191952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.893343    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:51.191952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.893405    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:30:51.191952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:51.176505    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:51.191952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:51.005242    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p\" (UID: \"8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"
Oct 31 13:30:51.322948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.811041    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:30:51.322948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.810924    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:30:51.421896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:51.408315    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:51.421896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:51.408519    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:51.567894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:51.463839    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"
Oct 31 13:30:51.567894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:50.811014    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:30:51.937327 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:51.931704    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:52.129006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:52.111437    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:52.130633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:52.130597    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:52.203530 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:52.188884    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:52.203530 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:52.189510    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:30:52.255893 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:52.251961    4428 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.128s"
Oct 31 13:30:52.926890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:52.926827    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs" event={"ID":"8e4b37cc-b081-4b6b-a71b-21b6cfc5766f","Type":"ContainerStarted","Data":"5763bc6c6de423f21c4d1bfabe97a4d5930a07c7f42f95884ea69bc4f2f20403"}
Oct 31 13:30:53.052132 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:52.279996    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:30:53.055085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:52.280046    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:53.068860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:52.763427    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:53.071840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:52.769144    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:30:53.186326 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.186266    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:30:53.189171 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.188451    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:53.191327 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.189710    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:53.226556 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.225132    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:53.287528 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:30:53.287469    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54553f5b_58e8_45da_a20c_bdd60e1dcfd6.slice/crio-4b40ef8be59b885389d176d71a073a3820c936c78f2523d505e6c5335e480854.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54553f5b_58e8_45da_a20c_bdd60e1dcfd6.slice/crio-conmon-4b40ef8be59b885389d176d71a073a3820c936c78f2523d505e6c5335e480854.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 13:30:53.421804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.421758    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:53.471964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.471283    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:30:53.487885 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.481110    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"
Oct 31 13:30:53.487885 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.481723    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"]
Oct 31 13:30:53.487885 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.485603    4428 cpu_manager.go:414] "containerMap: removing container" podUID="54553f5b-58e8-45da-a20c-bdd60e1dcfd6" containerName="revoke-keys"
Oct 31 13:30:53.487885 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.485623    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54553f5b-58e8-45da-a20c-bdd60e1dcfd6" containerName="revoke-keys"
Oct 31 13:30:53.487885 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.485818    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="54553f5b-58e8-45da-a20c-bdd60e1dcfd6" containerName="revoke-keys"
Oct 31 13:30:53.496728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.494058    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"]
Oct 31 13:30:53.496728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.494102    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"]
Oct 31 13:30:53.496728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.494152    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:30:53.496728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.494183    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:30:53.496728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.494213    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:30:53.496728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.494241    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:30:53.496728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.494270    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:30:53.496728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.494300    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:30:53.496728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.495286    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:30:53.500928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.500855    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs" event={"ID":"8e4b37cc-b081-4b6b-a71b-21b6cfc5766f","Type":"ContainerStarted","Data":"cd3fa0b04ef4a41c661bac2eebe6bf29a9709e79eb09673ae35ad0641c8d63a5"}
Oct 31 13:30:53.501753 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.501733    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"
Oct 31 13:30:53.504744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.504626    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c7b990ba-serving-cert\""
Oct 31 13:30:53.506589 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.505161    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh" event={"ID":"54553f5b-58e8-45da-a20c-bdd60e1dcfd6","Type":"ContainerDied","Data":"4bc55e0962c3197ca5afb50cfac47a1a65a62093c4a5862e53f202dae08029e2"}
Oct 31 13:30:53.506589 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.505228    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bc55e0962c3197ca5afb50cfac47a1a65a62093c4a5862e53f202dae08029e2"
Oct 31 13:30:53.506589 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.505301    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"
Oct 31 13:30:53.507299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.507270    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:30:53.511820 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.508257    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:30:53.526339 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.526313    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-maas-api-xks-ca\") pod \"54553f5b-58e8-45da-a20c-bdd60e1dcfd6\" (UID: \"54553f5b-58e8-45da-a20c-bdd60e1dcfd6\") "
Oct 31 13:30:53.526455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.526424    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-maas-api-openshift-ca\") pod \"54553f5b-58e8-45da-a20c-bdd60e1dcfd6\" (UID: \"54553f5b-58e8-45da-a20c-bdd60e1dcfd6\") "
Oct 31 13:30:53.526536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.526460    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-kube-api-access-rr5db\" (UniqueName: \"kubernetes.io/projected/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-kube-api-access-rr5db\") pod \"54553f5b-58e8-45da-a20c-bdd60e1dcfd6\" (UID: \"54553f5b-58e8-45da-a20c-bdd60e1dcfd6\") "
Oct 31 13:30:53.526833 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.526810    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-maas-api-openshift-ca" pod "54553f5b-58e8-45da-a20c-bdd60e1dcfd6" (UID: "54553f5b-58e8-45da-a20c-bdd60e1dcfd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:30:53.526964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.526921    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b454c7dc-1585-4c9a-9d89-e47285739d60-metrics-certs\") pod \"maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw\" (UID: \"b454c7dc-1585-4c9a-9d89-e47285739d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:30:53.527154 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.527129    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt8jj\" (UniqueName: \"kubernetes.io/projected/b454c7dc-1585-4c9a-9d89-e47285739d60-kube-api-access-rt8jj\") pod \"maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw\" (UID: \"b454c7dc-1585-4c9a-9d89-e47285739d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:30:53.527358 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.527338    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b454c7dc-1585-4c9a-9d89-e47285739d60-maas-api-tls\") pod \"maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw\" (UID: \"b454c7dc-1585-4c9a-9d89-e47285739d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:30:53.527441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.527411    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b454c7dc-1585-4c9a-9d89-e47285739d60-openshift-service-ca\") pod \"maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw\" (UID: \"b454c7dc-1585-4c9a-9d89-e47285739d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:30:53.527816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.527793    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:53.529074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.529045    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-maas-api-xks-ca" pod "54553f5b-58e8-45da-a20c-bdd60e1dcfd6" (UID: "54553f5b-58e8-45da-a20c-bdd60e1dcfd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:30:53.532052 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.532017    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-kube-api-access-rr5db" pod "54553f5b-58e8-45da-a20c-bdd60e1dcfd6" (UID: "54553f5b-58e8-45da-a20c-bdd60e1dcfd6"). InnerVolumeSpecName "kube-api-access-rr5db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:30:53.601568 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.601510    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs" podStartSLOduration=11.601489256 podStartE2EDuration="11.601489256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:53.595166592 +0000 UTC m=+1650.993515229" watchObservedRunningTime="2026-10-31 13:30:53.601489256 +0000 UTC m=+1650.999837881"
Oct 31 13:30:53.630185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.630150    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b454c7dc-1585-4c9a-9d89-e47285739d60-maas-api-tls\") pod \"maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw\" (UID: \"b454c7dc-1585-4c9a-9d89-e47285739d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:30:53.630346 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.630209    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b454c7dc-1585-4c9a-9d89-e47285739d60-openshift-service-ca\") pod \"maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw\" (UID: \"b454c7dc-1585-4c9a-9d89-e47285739d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:30:53.630476 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.630458    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b454c7dc-1585-4c9a-9d89-e47285739d60-metrics-certs\") pod \"maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw\" (UID: \"b454c7dc-1585-4c9a-9d89-e47285739d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:30:53.630689 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.630670    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt8jj\" (UniqueName: \"kubernetes.io/projected/b454c7dc-1585-4c9a-9d89-e47285739d60-kube-api-access-rt8jj\") pod \"maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw\" (UID: \"b454c7dc-1585-4c9a-9d89-e47285739d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:30:53.630797 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.630780    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr5db\" (UniqueName: \"kubernetes.io/projected/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-kube-api-access-rr5db\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:53.630855 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.630805    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54553f5b-58e8-45da-a20c-bdd60e1dcfd6-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:53.631013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.630985    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b454c7dc-1585-4c9a-9d89-e47285739d60-openshift-service-ca\") pod \"maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw\" (UID: \"b454c7dc-1585-4c9a-9d89-e47285739d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:30:53.633711 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.633688    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b454c7dc-1585-4c9a-9d89-e47285739d60-metrics-certs\") pod \"maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw\" (UID: \"b454c7dc-1585-4c9a-9d89-e47285739d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:30:53.634356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.634333    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b454c7dc-1585-4c9a-9d89-e47285739d60-maas-api-tls\") pod \"maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw\" (UID: \"b454c7dc-1585-4c9a-9d89-e47285739d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:30:53.652361 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.652334    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt8jj\" (UniqueName: \"kubernetes.io/projected/b454c7dc-1585-4c9a-9d89-e47285739d60-kube-api-access-rt8jj\") pod \"maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw\" (UID: \"b454c7dc-1585-4c9a-9d89-e47285739d60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:30:53.703307 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.703272    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"]
Oct 31 13:30:53.703447 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:30:53.703420    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d02f56a_71e6_4ca8_b1e2_e24b7b21f5fa.slice/crio-d1328130802ff1dbe4f2b96c2e4dbe7a10f656f13682ebeb9ea57dd6f989a1ab WatchSource:0}: Error finding container d1328130802ff1dbe4f2b96c2e4dbe7a10f656f13682ebeb9ea57dd6f989a1ab: Status 404 returned error can't find the container with id d1328130802ff1dbe4f2b96c2e4dbe7a10f656f13682ebeb9ea57dd6f989a1ab
Oct 31 13:30:53.819943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:53.819916    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:30:54.255262 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:54.255230    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:54.431543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:54.430132    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"]
Oct 31 13:30:54.436207 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:30:54.436176    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb454c7dc_1585_4c9a_9d89_e47285739d60.slice/crio-7fb7f1894f95ad8f2ad40ad3d208c3feef7cc84d5f078e59ff3df019b25552e1 WatchSource:0}: Error finding container 7fb7f1894f95ad8f2ad40ad3d208c3feef7cc84d5f078e59ff3df019b25552e1: Status 404 returned error can't find the container with id 7fb7f1894f95ad8f2ad40ad3d208c3feef7cc84d5f078e59ff3df019b25552e1
Oct 31 13:30:54.514949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:54.514855    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p" event={"ID":"8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa","Type":"ContainerStarted","Data":"8feaab8151f2a2c4d9fb1c92fd7f395c2ac44883e331cca2711397482a3b6322"}
Oct 31 13:30:54.514949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:54.514927    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p" event={"ID":"8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa","Type":"ContainerStarted","Data":"d1328130802ff1dbe4f2b96c2e4dbe7a10f656f13682ebeb9ea57dd6f989a1ab"}
Oct 31 13:30:54.515130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:54.514975    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"
Oct 31 13:30:54.516456 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:54.516432    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw" event={"ID":"b454c7dc-1585-4c9a-9d89-e47285739d60","Type":"ContainerStarted","Data":"7fb7f1894f95ad8f2ad40ad3d208c3feef7cc84d5f078e59ff3df019b25552e1"}
Oct 31 13:30:54.541596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:54.541553    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p" podStartSLOduration=6.541540667 podStartE2EDuration="6.541540667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:54.539351054 +0000 UTC m=+1651.937699678" watchObservedRunningTime="2026-10-31 13:30:54.541540667 +0000 UTC m=+1651.939889291"
Oct 31 13:30:55.443544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:55.443442    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:55.528677 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:55.528637    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw" event={"ID":"b454c7dc-1585-4c9a-9d89-e47285739d60","Type":"ContainerStarted","Data":"e0463ddeca19d8a4d331e3eca6f640d209823a6edb6f5434cf8a04331f6d099b"}
Oct 31 13:30:55.528938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:55.528913    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:30:55.661894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:55.657847    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:55.661894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:55.658716    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw" podStartSLOduration=5.115193528 podStartE2EDuration="5.658699191s" totalImagesPullingTime="543.505663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:55.562959505 +0000 UTC m=+1652.961308129" watchObservedRunningTime="2026-10-31 13:30:55.658699191 +0000 UTC m=+1653.057047815"
Oct 31 13:30:56.528936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:56.522111    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:56.849617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:56.849525    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:56.927633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:56.927441    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"]
Oct 31 13:30:56.932510 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:56.932489    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"
Oct 31 13:30:56.943614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:56.943593    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"]
Oct 31 13:30:56.996892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:56.988756    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a66ae960-dockercfg-ql9xg\""
Oct 31 13:30:56.996892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:56.988998    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a66ae960\""
Oct 31 13:30:57.098896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:57.062209    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1004aeb3-63d7-48e5-ac93-59eea0a4c66a-plugins-config-volume\") pod \"payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v\" (UID: \"1004aeb3-63d7-48e5-ac93-59eea0a4c66a\") " pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"
Oct 31 13:30:57.098896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:57.062364    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gps56\" (UniqueName: \"kubernetes.io/projected/1004aeb3-63d7-48e5-ac93-59eea0a4c66a-kube-api-access-gps56\") pod \"payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v\" (UID: \"1004aeb3-63d7-48e5-ac93-59eea0a4c66a\") " pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"
Oct 31 13:30:57.127267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:57.125847    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:57.168892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:57.166212    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1004aeb3-63d7-48e5-ac93-59eea0a4c66a-plugins-config-volume\") pod \"payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v\" (UID: \"1004aeb3-63d7-48e5-ac93-59eea0a4c66a\") " pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"
Oct 31 13:30:57.168892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:57.166358    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gps56\" (UniqueName: \"kubernetes.io/projected/1004aeb3-63d7-48e5-ac93-59eea0a4c66a-kube-api-access-gps56\") pod \"payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v\" (UID: \"1004aeb3-63d7-48e5-ac93-59eea0a4c66a\") " pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"
Oct 31 13:30:57.168892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:57.168104    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1004aeb3-63d7-48e5-ac93-59eea0a4c66a-plugins-config-volume\") pod \"payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v\" (UID: \"1004aeb3-63d7-48e5-ac93-59eea0a4c66a\") " pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"
Oct 31 13:30:57.201917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:57.192992    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gps56\" (UniqueName: \"kubernetes.io/projected/1004aeb3-63d7-48e5-ac93-59eea0a4c66a-kube-api-access-gps56\") pod \"payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v\" (UID: \"1004aeb3-63d7-48e5-ac93-59eea0a4c66a\") " pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"
Oct 31 13:30:57.264895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:57.262250    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"
Oct 31 13:30:57.728900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:57.728847    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"]
Oct 31 13:30:57.731758 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:30:57.731727    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1004aeb3_63d7_48e5_ac93_59eea0a4c66a.slice/crio-493c21eb622a4391472925eced646a9a82829200b87e99ad276165cd2dd6199d WatchSource:0}: Error finding container 493c21eb622a4391472925eced646a9a82829200b87e99ad276165cd2dd6199d: Status 404 returned error can't find the container with id 493c21eb622a4391472925eced646a9a82829200b87e99ad276165cd2dd6199d
Oct 31 13:30:57.741854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:57.741821    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:58.032348 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:58.032269    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:58.669850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:58.664566    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v" event={"ID":"1004aeb3-63d7-48e5-ac93-59eea0a4c66a","Type":"ContainerStarted","Data":"64ac1ca52d47d70e1cc0b7a36bb90a28b260fcb1724e0900e19e54bd26a3fbd5"}
Oct 31 13:30:58.669850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:58.664635    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v" event={"ID":"1004aeb3-63d7-48e5-ac93-59eea0a4c66a","Type":"ContainerStarted","Data":"493c21eb622a4391472925eced646a9a82829200b87e99ad276165cd2dd6199d"}
Oct 31 13:30:58.669850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:58.664913    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"
Oct 31 13:30:58.727559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:58.727323    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v" podStartSLOduration=2.727304507 podStartE2EDuration="2.727304507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:58.712004586 +0000 UTC m=+1656.110353231" watchObservedRunningTime="2026-10-31 13:30:58.727304507 +0000 UTC m=+1656.125653130"
Oct 31 13:30:58.916893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:58.916254    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:59.153371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:59.153335    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:30:59.722883 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:30:59.722784    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:00.037314 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:00.037239    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:00.312328 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:00.312246    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:00.526920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:00.515523    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:00.854227 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:00.854147    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:01.112226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.112127    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:01.316963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.316927    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"]
Oct 31 13:31:01.320905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.320857    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.323598 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.323572    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"]
Oct 31 13:31:01.334748 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.334643    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:01.334934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.334801    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-dff70365-openshift-default-dockercfg-zt7pd\""
Oct 31 13:31:01.450021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.439828    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-workload-certs\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.450021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.439941    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-credential-socket\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.450021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.440013    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af8ebab8-5e74-42e6-ac89-b00164117531-istiod-ca-cert\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.450021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.440056    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-istio-envoy\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.450021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.440166    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-workload-socket\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.450021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.440249    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-istio-data\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.450021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.440284    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sktq9\" (UniqueName: \"kubernetes.io/projected/af8ebab8-5e74-42e6-ac89-b00164117531-kube-api-access-sktq9\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.450021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.440310    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af8ebab8-5e74-42e6-ac89-b00164117531-istio-podinfo\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.450021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.440329    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af8ebab8-5e74-42e6-ac89-b00164117531-istio-token\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.519331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.519295    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:01.543469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.543425    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-workload-certs\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.544274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.544249    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-credential-socket\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.544492 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.544475    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af8ebab8-5e74-42e6-ac89-b00164117531-istiod-ca-cert\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.544616 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.544599    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-istio-envoy\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.544848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.544831    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-workload-socket\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.554488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.554463    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-istio-data\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.555345 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.555324    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktq9\" (UniqueName: \"kubernetes.io/projected/af8ebab8-5e74-42e6-ac89-b00164117531-kube-api-access-sktq9\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.555481 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.555466    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af8ebab8-5e74-42e6-ac89-b00164117531-istio-podinfo\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.555637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.555619    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af8ebab8-5e74-42e6-ac89-b00164117531-istio-token\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.555933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.545264    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-credential-socket\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.557045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.555262    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-workload-socket\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.568821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.554029    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af8ebab8-5e74-42e6-ac89-b00164117531-istiod-ca-cert\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.569075 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.544134    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-workload-certs\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.569227 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.554949    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-istio-data\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.607991 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.607953    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sktq9\" (UniqueName: \"kubernetes.io/projected/af8ebab8-5e74-42e6-ac89-b00164117531-kube-api-access-sktq9\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.619943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.615685    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af8ebab8-5e74-42e6-ac89-b00164117531-istio-token\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.636930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.636846    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-istio-envoy\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.647550 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.646307    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af8ebab8-5e74-42e6-ac89-b00164117531-istio-podinfo\") pod \"e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") " pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.647721 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.647565    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:01.663335 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:01.652079    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:31:02.182009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:02.181914    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"]
Oct 31 13:31:02.213016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:02.212928    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:02.220050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:02.219305    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:31:02.220050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:02.219421    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:31:02.220050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:02.219465    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:31:03.164893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:03.164769    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:03.200284 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:03.198228    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw" event={"ID":"af8ebab8-5e74-42e6-ac89-b00164117531","Type":"ContainerStarted","Data":"7f18665c48ce6dcb1a5cfc6aed48591ed9c96d5590fe3b22837cc579ca257910"}
Oct 31 13:31:03.200284 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:03.198293    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw" event={"ID":"af8ebab8-5e74-42e6-ac89-b00164117531","Type":"ContainerStarted","Data":"19354a81543c35af1c77f9de420670fb112f64d9aeaf090dbb68656f6ec2f6fb"}
Oct 31 13:31:03.257089 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:03.249811    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw" podStartSLOduration=2.249791847 podStartE2EDuration="2.249791847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:03.249176114 +0000 UTC m=+1660.647524752" watchObservedRunningTime="2026-10-31 13:31:03.249791847 +0000 UTC m=+1660.648140471"
Oct 31 13:31:03.648944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:03.648912    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:03.655428 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:03.655405    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:04.398126 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:04.394594    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:04.398126 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:04.395812    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:04.398126 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:04.396017    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:31:04.606231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:04.606045    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:04.885600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:04.885566    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:05.054572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:05.054541    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:05.532100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:05.532072    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"
Oct 31 13:31:05.532733 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:05.532706    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"
Oct 31 13:31:05.785818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:05.785722    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:05.878976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:05.878947    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"]
Oct 31 13:31:05.894450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:05.894411    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"
Oct 31 13:31:05.898952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:05.898929    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a8c1600c\""
Oct 31 13:31:05.902016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:05.901987    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"]
Oct 31 13:31:06.000540 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:06.000509    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:06.026718 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:06.026691    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm\" (UID: \"60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"
Oct 31 13:31:06.026859 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:06.026734    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8wh7\" (UniqueName: \"kubernetes.io/projected/60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2-kube-api-access-r8wh7\") pod \"payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm\" (UID: \"60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"
Oct 31 13:31:06.143107 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:06.143012    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm\" (UID: \"60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"
Oct 31 13:31:06.168223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:06.168194    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8wh7\" (UniqueName: \"kubernetes.io/projected/60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2-kube-api-access-r8wh7\") pod \"payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm\" (UID: \"60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"
Oct 31 13:31:06.196291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:06.166454    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm\" (UID: \"60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"
Oct 31 13:31:06.207901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:06.206613    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8wh7\" (UniqueName: \"kubernetes.io/projected/60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2-kube-api-access-r8wh7\") pod \"payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm\" (UID: \"60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"
Oct 31 13:31:06.217256 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:06.217221    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"
Oct 31 13:31:06.764771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:06.764747    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"]
Oct 31 13:31:06.836567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:06.836537    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:07.016340 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:07.016288    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:07.202489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:07.202405    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:07.736996 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:07.736960    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm" event={"ID":"60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2","Type":"ContainerStarted","Data":"961e49feda722eb2f4b32938c5022dd3c6f7f0d2f3550c762e4345fb707e5d65"}
Oct 31 13:31:07.737269 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:07.737247    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm" event={"ID":"60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2","Type":"ContainerStarted","Data":"cd2bef93767654a71c6d87e44c4c4102e6e4f54e7e32f3338ee42e774e671edd"}
Oct 31 13:31:07.737393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:07.737381    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"
Oct 31 13:31:07.787964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:07.787530    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm" podStartSLOduration=2.787507287 podStartE2EDuration="2.787507287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:07.769274532 +0000 UTC m=+1665.167623157" watchObservedRunningTime="2026-10-31 13:31:07.787507287 +0000 UTC m=+1665.185855913"
Oct 31 13:31:07.869158 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:07.869125    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"]
Oct 31 13:31:07.874261 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:07.874241    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"
Oct 31 13:31:07.882279 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:07.879535    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:31:07.882279 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:07.879766    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:31:07.883398 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:07.883374    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"]
Oct 31 13:31:08.000949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:07.982243    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt\" (UID: \"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"
Oct 31 13:31:08.000949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:07.982346    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-httkz\" (UniqueName: \"kubernetes.io/projected/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-kube-api-access-httkz\") pod \"maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt\" (UID: \"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"
Oct 31 13:31:08.000949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:07.982514    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt\" (UID: \"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"
Oct 31 13:31:08.088966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.088925    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt\" (UID: \"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"
Oct 31 13:31:08.089191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.089174    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt\" (UID: \"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"
Oct 31 13:31:08.089272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.089244    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-httkz\" (UniqueName: \"kubernetes.io/projected/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-kube-api-access-httkz\") pod \"maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt\" (UID: \"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"
Oct 31 13:31:08.089847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.089823    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt\" (UID: \"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"
Oct 31 13:31:08.091486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.091467    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt\" (UID: \"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"
Oct 31 13:31:08.105568 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.105540    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-httkz\" (UniqueName: \"kubernetes.io/projected/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-kube-api-access-httkz\") pod \"maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt\" (UID: \"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"
Oct 31 13:31:08.125496 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.125474    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"]
Oct 31 13:31:08.130478 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.130463    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:31:08.139043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.139024    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a66ae960-serving-cert\""
Oct 31 13:31:08.184823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.184790    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"]
Oct 31 13:31:08.198595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.198568    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"
Oct 31 13:31:08.292148 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.292118    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e799f52e-4fcc-47be-b42c-d43976f7660f-maas-api-tls\") pod \"maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb\" (UID: \"e799f52e-4fcc-47be-b42c-d43976f7660f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:31:08.292312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.292162    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5sjd\" (UniqueName: \"kubernetes.io/projected/e799f52e-4fcc-47be-b42c-d43976f7660f-kube-api-access-z5sjd\") pod \"maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb\" (UID: \"e799f52e-4fcc-47be-b42c-d43976f7660f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:31:08.292414 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.292397    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e799f52e-4fcc-47be-b42c-d43976f7660f-openshift-service-ca\") pod \"maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb\" (UID: \"e799f52e-4fcc-47be-b42c-d43976f7660f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:31:08.292546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.292531    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e799f52e-4fcc-47be-b42c-d43976f7660f-metrics-certs\") pod \"maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb\" (UID: \"e799f52e-4fcc-47be-b42c-d43976f7660f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:31:08.342787 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.342419    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:08.394820 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.394767    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e799f52e-4fcc-47be-b42c-d43976f7660f-openshift-service-ca\") pod \"maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb\" (UID: \"e799f52e-4fcc-47be-b42c-d43976f7660f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:31:08.395172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.395128    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e799f52e-4fcc-47be-b42c-d43976f7660f-metrics-certs\") pod \"maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb\" (UID: \"e799f52e-4fcc-47be-b42c-d43976f7660f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:31:08.395515 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.395473    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e799f52e-4fcc-47be-b42c-d43976f7660f-maas-api-tls\") pod \"maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb\" (UID: \"e799f52e-4fcc-47be-b42c-d43976f7660f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:31:08.395767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.395661    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5sjd\" (UniqueName: \"kubernetes.io/projected/e799f52e-4fcc-47be-b42c-d43976f7660f-kube-api-access-z5sjd\") pod \"maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb\" (UID: \"e799f52e-4fcc-47be-b42c-d43976f7660f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:31:08.397208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.397160    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e799f52e-4fcc-47be-b42c-d43976f7660f-openshift-service-ca\") pod \"maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb\" (UID: \"e799f52e-4fcc-47be-b42c-d43976f7660f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:31:08.578460 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.578351    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"]
Oct 31 13:31:08.579526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.579503    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e799f52e-4fcc-47be-b42c-d43976f7660f-maas-api-tls\") pod \"maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb\" (UID: \"e799f52e-4fcc-47be-b42c-d43976f7660f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:31:08.580130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.580104    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e799f52e-4fcc-47be-b42c-d43976f7660f-metrics-certs\") pod \"maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb\" (UID: \"e799f52e-4fcc-47be-b42c-d43976f7660f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:31:08.584220 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:31:08.584192    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711e30cb_0fbe_41bc_9cd7_a1e9c1137eb2.slice/crio-53fa0b1d9fac34ecc154277a8b5ac5c3dde80601173c8455990c085f358a1abb WatchSource:0}: Error finding container 53fa0b1d9fac34ecc154277a8b5ac5c3dde80601173c8455990c085f358a1abb: Status 404 returned error can't find the container with id 53fa0b1d9fac34ecc154277a8b5ac5c3dde80601173c8455990c085f358a1abb
Oct 31 13:31:08.584350 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.584334    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5sjd\" (UniqueName: \"kubernetes.io/projected/e799f52e-4fcc-47be-b42c-d43976f7660f-kube-api-access-z5sjd\") pod \"maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb\" (UID: \"e799f52e-4fcc-47be-b42c-d43976f7660f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:31:08.606731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.606699    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:08.755422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.755137    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:31:08.869921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:08.864684    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt" event={"ID":"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2","Type":"ContainerStarted","Data":"53fa0b1d9fac34ecc154277a8b5ac5c3dde80601173c8455990c085f358a1abb"}
Oct 31 13:31:09.374590 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:31:09.374562    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode799f52e_4fcc_47be_b42c_d43976f7660f.slice/crio-161b06f5ef705e7ea82443fdc4528f59ca86d420593d44d69b7bc70f506fd3e6 WatchSource:0}: Error finding container 161b06f5ef705e7ea82443fdc4528f59ca86d420593d44d69b7bc70f506fd3e6: Status 404 returned error can't find the container with id 161b06f5ef705e7ea82443fdc4528f59ca86d420593d44d69b7bc70f506fd3e6
Oct 31 13:31:09.375747 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:09.375719    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"]
Oct 31 13:31:09.481036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:09.480944    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:09.669776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:09.669696    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:09.687610 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:09.687582    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"
Oct 31 13:31:09.873672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:09.873645    4428 generic.go:356] "Generic (PLEG): container finished" podID="711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2" containerID="fb9c0312662263e78e83ea8625a142b46e9f37719c7c1675f4f47ad85d907151" exitCode=0
Oct 31 13:31:09.874100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:09.873672    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt" event={"ID":"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2","Type":"ContainerDied","Data":"fb9c0312662263e78e83ea8625a142b46e9f37719c7c1675f4f47ad85d907151"}
Oct 31 13:31:09.875255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:09.875234    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb" event={"ID":"e799f52e-4fcc-47be-b42c-d43976f7660f","Type":"ContainerStarted","Data":"161b06f5ef705e7ea82443fdc4528f59ca86d420593d44d69b7bc70f506fd3e6"}
Oct 31 13:31:10.456892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:10.452995    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:10.885267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:10.885227    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb" event={"ID":"e799f52e-4fcc-47be-b42c-d43976f7660f","Type":"ContainerStarted","Data":"4b81cf0cd81b0f911ab21567935b50ca649be412b4e9e14fcf506e3ee56227fa"}
Oct 31 13:31:10.885676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:10.885287    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:31:10.911533 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:10.911485    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb" podStartSLOduration=2.414865403 podStartE2EDuration="2.911473001s" totalImagesPullingTime="496.607598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:10.906000488 +0000 UTC m=+1668.304349113" watchObservedRunningTime="2026-10-31 13:31:10.911473001 +0000 UTC m=+1668.309821625"
Oct 31 13:31:11.206613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:11.206584    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"
Oct 31 13:31:11.296360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:11.296329    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-maas-api-openshift-ca\") pod \"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2\" (UID: \"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2\") "
Oct 31 13:31:11.296524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:11.296457    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-maas-api-xks-ca\") pod \"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2\" (UID: \"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2\") "
Oct 31 13:31:11.296589 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:11.296572    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-kube-api-access-httkz\" (UniqueName: \"kubernetes.io/projected/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-kube-api-access-httkz\") pod \"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2\" (UID: \"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2\") "
Oct 31 13:31:11.296676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:11.296650    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-maas-api-openshift-ca" pod "711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2" (UID: "711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:31:11.297011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:11.296996    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:11.298807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:11.298778    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-kube-api-access-httkz" pod "711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2" (UID: "711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2"). InnerVolumeSpecName "kube-api-access-httkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:31:11.298807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:11.298797    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-maas-api-xks-ca" pod "711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2" (UID: "711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:31:11.399127 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:11.399044    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-httkz\" (UniqueName: \"kubernetes.io/projected/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-kube-api-access-httkz\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:11.399127 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:11.399090    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:11.570117 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:11.569948    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:11.896794 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:11.896765    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"
Oct 31 13:31:11.897194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:11.896802    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt" event={"ID":"711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2","Type":"ContainerDied","Data":"53fa0b1d9fac34ecc154277a8b5ac5c3dde80601173c8455990c085f358a1abb"}
Oct 31 13:31:11.897194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:11.896857    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53fa0b1d9fac34ecc154277a8b5ac5c3dde80601173c8455990c085f358a1abb"
Oct 31 13:31:12.499998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:12.494519    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:12.856933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:12.856847    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:13.758892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:13.755954    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:14.143258 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.143181    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"]
Oct 31 13:31:14.143976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.143960    4428 cpu_manager.go:414] "containerMap: removing container" podUID="711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2" containerName="revoke-keys"
Oct 31 13:31:14.143976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.143977    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2" containerName="revoke-keys"
Oct 31 13:31:14.144142 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.144132    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2" containerName="revoke-keys"
Oct 31 13:31:14.148020 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.147999    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"
Oct 31 13:31:14.154892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.154659    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:31:14.154892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.154755    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:31:14.177131 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.177104    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"]
Oct 31 13:31:14.271590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.271556    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c19b695c-87c3-41b8-bb49-4a671676a4e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8439b78a-c59kn\" (UID: \"c19b695c-87c3-41b8-bb49-4a671676a4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"
Oct 31 13:31:14.271749 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.271626    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c19b695c-87c3-41b8-bb49-4a671676a4e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8439b78a-c59kn\" (UID: \"c19b695c-87c3-41b8-bb49-4a671676a4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"
Oct 31 13:31:14.271805 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.271781    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2crb\" (UniqueName: \"kubernetes.io/projected/c19b695c-87c3-41b8-bb49-4a671676a4e4-kube-api-access-p2crb\") pod \"maas-api-revoke-keys-e2e-ait-8439b78a-c59kn\" (UID: \"c19b695c-87c3-41b8-bb49-4a671676a4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"
Oct 31 13:31:14.373641 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.373606    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2crb\" (UniqueName: \"kubernetes.io/projected/c19b695c-87c3-41b8-bb49-4a671676a4e4-kube-api-access-p2crb\") pod \"maas-api-revoke-keys-e2e-ait-8439b78a-c59kn\" (UID: \"c19b695c-87c3-41b8-bb49-4a671676a4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"
Oct 31 13:31:14.373793 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.373757    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c19b695c-87c3-41b8-bb49-4a671676a4e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8439b78a-c59kn\" (UID: \"c19b695c-87c3-41b8-bb49-4a671676a4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"
Oct 31 13:31:14.373835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.373810    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c19b695c-87c3-41b8-bb49-4a671676a4e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8439b78a-c59kn\" (UID: \"c19b695c-87c3-41b8-bb49-4a671676a4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"
Oct 31 13:31:14.374445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.374421    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c19b695c-87c3-41b8-bb49-4a671676a4e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8439b78a-c59kn\" (UID: \"c19b695c-87c3-41b8-bb49-4a671676a4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"
Oct 31 13:31:14.376343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.376317    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c19b695c-87c3-41b8-bb49-4a671676a4e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8439b78a-c59kn\" (UID: \"c19b695c-87c3-41b8-bb49-4a671676a4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"
Oct 31 13:31:14.387451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.387425    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2crb\" (UniqueName: \"kubernetes.io/projected/c19b695c-87c3-41b8-bb49-4a671676a4e4-kube-api-access-p2crb\") pod \"maas-api-revoke-keys-e2e-ait-8439b78a-c59kn\" (UID: \"c19b695c-87c3-41b8-bb49-4a671676a4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"
Oct 31 13:31:14.468192 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.468161    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"
Oct 31 13:31:14.764800 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:31:14.764771    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19b695c_87c3_41b8_bb49_4a671676a4e4.slice/crio-0222e09109ea04608153afd4c59031ab4817823d981f7c1e22dae7604ecbd004 WatchSource:0}: Error finding container 0222e09109ea04608153afd4c59031ab4817823d981f7c1e22dae7604ecbd004: Status 404 returned error can't find the container with id 0222e09109ea04608153afd4c59031ab4817823d981f7c1e22dae7604ecbd004
Oct 31 13:31:14.765712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.765685    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"]
Oct 31 13:31:14.878738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.869294    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:14.943056 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:14.938857    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn" event={"ID":"c19b695c-87c3-41b8-bb49-4a671676a4e4","Type":"ContainerStarted","Data":"0222e09109ea04608153afd4c59031ab4817823d981f7c1e22dae7604ecbd004"}
Oct 31 13:31:16.025587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:16.017889    4428 generic.go:356] "Generic (PLEG): container finished" podID="c19b695c-87c3-41b8-bb49-4a671676a4e4" containerID="304f599b37ed35079b998f1616714763a8f0801ff8119e8c5ea44f5ba36bd713" exitCode=0
Oct 31 13:31:16.025587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:16.018155    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn" event={"ID":"c19b695c-87c3-41b8-bb49-4a671676a4e4","Type":"ContainerDied","Data":"304f599b37ed35079b998f1616714763a8f0801ff8119e8c5ea44f5ba36bd713"}
Oct 31 13:31:16.236896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:16.231004    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:16.972893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:16.966167    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:31:18.576898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:18.550669    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"
Oct 31 13:31:18.722951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:18.696709    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:18.831857 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:18.831762    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:19.041075 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.041039    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c19b695c-87c3-41b8-bb49-4a671676a4e4-kube-api-access-p2crb\" (UniqueName: \"kubernetes.io/projected/c19b695c-87c3-41b8-bb49-4a671676a4e4-kube-api-access-p2crb\") pod \"c19b695c-87c3-41b8-bb49-4a671676a4e4\" (UID: \"c19b695c-87c3-41b8-bb49-4a671676a4e4\") "
Oct 31 13:31:19.221040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.090340    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:19.221040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.180204    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:31:19.221040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.180705    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:31:19.222653 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.222614    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c19b695c-87c3-41b8-bb49-4a671676a4e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c19b695c-87c3-41b8-bb49-4a671676a4e4-maas-api-xks-ca\") pod \"c19b695c-87c3-41b8-bb49-4a671676a4e4\" (UID: \"c19b695c-87c3-41b8-bb49-4a671676a4e4\") "
Oct 31 13:31:19.225197 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.222857    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c19b695c-87c3-41b8-bb49-4a671676a4e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c19b695c-87c3-41b8-bb49-4a671676a4e4-maas-api-openshift-ca\") pod \"c19b695c-87c3-41b8-bb49-4a671676a4e4\" (UID: \"c19b695c-87c3-41b8-bb49-4a671676a4e4\") "
Oct 31 13:31:19.246482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.246434    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19b695c-87c3-41b8-bb49-4a671676a4e4-maas-api-openshift-ca" pod "c19b695c-87c3-41b8-bb49-4a671676a4e4" (UID: "c19b695c-87c3-41b8-bb49-4a671676a4e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:31:19.285410 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.285373    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:31:19.315052 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.315011    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld" podUID="6002eb05-9237-48f2-972a-448248703ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:19.451901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.451255    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:31:19.516946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.491134    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c19b695c-87c3-41b8-bb49-4a671676a4e4-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:19.516946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.514365    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:19.536998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.521226    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19b695c-87c3-41b8-bb49-4a671676a4e4-maas-api-xks-ca" pod "c19b695c-87c3-41b8-bb49-4a671676a4e4" (UID: "c19b695c-87c3-41b8-bb49-4a671676a4e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:31:19.536998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.521566    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"
Oct 31 13:31:19.536998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.523380    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn" event={"ID":"c19b695c-87c3-41b8-bb49-4a671676a4e4","Type":"ContainerDied","Data":"0222e09109ea04608153afd4c59031ab4817823d981f7c1e22dae7604ecbd004"}
Oct 31 13:31:19.536998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.523431    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0222e09109ea04608153afd4c59031ab4817823d981f7c1e22dae7604ecbd004"
Oct 31 13:31:19.536998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.523535    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"
Oct 31 13:31:19.643897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.609446    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19b695c-87c3-41b8-bb49-4a671676a4e4-kube-api-access-p2crb" pod "c19b695c-87c3-41b8-bb49-4a671676a4e4" (UID: "c19b695c-87c3-41b8-bb49-4a671676a4e4"). InnerVolumeSpecName "kube-api-access-p2crb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:31:19.643897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.610925    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2crb\" (UniqueName: \"kubernetes.io/projected/c19b695c-87c3-41b8-bb49-4a671676a4e4-kube-api-access-p2crb\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:19.643897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.610951    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c19b695c-87c3-41b8-bb49-4a671676a4e4-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:19.853603 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.852951    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:19.940896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.940102    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:19.988025 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.987981    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:20.001933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:19.992023    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Oct 31 13:31:20.014911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:20.014196    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:31:20.032937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:20.032909    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:31:20.047430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:20.047406    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:31:20.048486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:20.046134    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:31:20.065903 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:31:20.059595    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:31:20.217903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:20.185131    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:20.217903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:20.185452    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:31:20.217903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:20.186011    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:31:20.217903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:20.186076    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:31:20.568948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:20.452999    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:20.568948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:20.453420    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:31:20.683895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:20.672367    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:20.952948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:20.939897    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:20.952948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:20.942166    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:20.952948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:20.943092    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:31:21.000662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:20.993031    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:21.058000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:21.053100    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:21.080140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:21.079366    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:31:21.087531 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:21.080515    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:21.197818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:21.197769    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:21.198491 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:21.198449    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:31:21.500893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:21.479806    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:31:21.744603 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:21.744574    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:31:21.782643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:21.782568    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:31:21.783852 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:21.525447    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:22.088951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:22.088240    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:31:22.115901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:22.097402    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:31:22.335510 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:22.198215    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:22.341059 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:22.227184    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:31:22.445903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:22.434345    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:31:22.629657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:22.629554    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h" podUID="a7c64ba1-f422-427b-80b8-1f464d32fafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:22.655354 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:22.408122    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:31:22.682060 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:22.628841    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:31:22.682399 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:22.682340    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:22.746931 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:22.365041    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:31:23.061977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:23.049625    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:31:23.188297 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:23.188253    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:23.189513 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:23.189396    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:31:23.245895 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:31:23.242202    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19b695c_87c3_41b8_bb49_4a671676a4e4.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:31:23.380111 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:23.380005    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:31:23.383302 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:31:23.383266    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19b695c_87c3_41b8_bb49_4a671676a4e4.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:31:23.573253 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:23.573212    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:31:23.574402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:23.574372    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:31:23.640264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:23.640145    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:31:23.723538 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:23.723496    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:31:23.805819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:23.725098    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:23.806404 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:23.723427    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-596gr" podUID="c20e3218-9e4a-43a1-b46e-050de8eb25c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:23.807680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:23.802684    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59" podUID="cd0ac048-d186-4f3c-929d-eb870afbc801" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Oct 31 13:31:23.808036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:23.802740    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:31:23.813126 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:23.812412    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg" podUID="5c86b906-a276-4e67-8e58-79a17118f5fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Oct 31 13:31:23.990349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:23.986211    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47" podUID="ea978050-20b0-4148-ab41-75844fe1b637" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:23.990349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:23.986894    4428 patch_prober.go:29] interesting pod/metrics-server-84cd8cdd9d-wh5xd container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:31:23.990349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:23.986946    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-84cd8cdd9d-wh5xd" podUID="b6592e99-9caf-418f-ac38-9f58bc8f672a" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:24.066933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:24.064154    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:31:24.195671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:24.195631    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:24.299381 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:24.299272    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:24.311028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:24.310982    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:31:24.311267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:24.311254    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:31:24.311699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:24.311682    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:31:24.311892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:24.311860    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:31:24.327197 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:24.327163    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"]
Oct 31 13:31:24.341937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:24.341092    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:31:24.695895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:24.662146    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:24.695895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:24.670167    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"]
Oct 31 13:31:24.695895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:24.670555    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:31:24.876975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:24.876925    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:31:24.877539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:24.877491    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:31:25.126733 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:25.126705    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:31:25.146662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:25.146626    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"]
Oct 31 13:31:25.227325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:25.227229    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:25.448962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:25.448925    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:25.632921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:25.632802    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:27.683721 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:27.683684    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:28.474401 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:28.474357    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"]
Oct 31 13:31:28.626835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:28.626799    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:28.801651 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:28.801537    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:29.328610 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:29.328561    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829" podUID="a5ef0238-0e04-42d5-93b2-625f05dbaeb2" containerName="payload-processing" containerID="cri-o://a7bf51823cc388bb4ea437997fe510787c89469904dc39820b98d3baaaf45aac" gracePeriod=25
Oct 31 13:31:29.488495 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:29.488298    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:29.671578 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:29.670625    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv" podUID="7e49afd3-26bd-493f-a807-ccd12c325d74" containerName="payload-pre-processing" containerID="cri-o://472047b68ae71da26ca3363fbcae4644c270047302a782e1ad1632d86f1588e6" gracePeriod=25
Oct 31 13:31:29.811516 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:29.811490    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"
Oct 31 13:31:29.855324 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:29.855287    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ef0238-0e04-42d5-93b2-625f05dbaeb2-plugins-config-volume" pod "a5ef0238-0e04-42d5-93b2-625f05dbaeb2" (UID: "a5ef0238-0e04-42d5-93b2-625f05dbaeb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:31:29.855495 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:29.855335    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ef0238-0e04-42d5-93b2-625f05dbaeb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ef0238-0e04-42d5-93b2-625f05dbaeb2-plugins-config-volume\") pod \"a5ef0238-0e04-42d5-93b2-625f05dbaeb2\" (UID: \"a5ef0238-0e04-42d5-93b2-625f05dbaeb2\") "
Oct 31 13:31:29.855495 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:29.855432    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ef0238-0e04-42d5-93b2-625f05dbaeb2-kube-api-access-l2tl2\" (UniqueName: \"kubernetes.io/projected/a5ef0238-0e04-42d5-93b2-625f05dbaeb2-kube-api-access-l2tl2\") pod \"a5ef0238-0e04-42d5-93b2-625f05dbaeb2\" (UID: \"a5ef0238-0e04-42d5-93b2-625f05dbaeb2\") "
Oct 31 13:31:29.857285 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:29.857184    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ef0238-0e04-42d5-93b2-625f05dbaeb2-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:29.858069 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:29.858042    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ef0238-0e04-42d5-93b2-625f05dbaeb2-kube-api-access-l2tl2" pod "a5ef0238-0e04-42d5-93b2-625f05dbaeb2" (UID: "a5ef0238-0e04-42d5-93b2-625f05dbaeb2"). InnerVolumeSpecName "kube-api-access-l2tl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:31:29.958772 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:29.958703    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2tl2\" (UniqueName: \"kubernetes.io/projected/a5ef0238-0e04-42d5-93b2-625f05dbaeb2-kube-api-access-l2tl2\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:29.969718 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:29.969692    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"
Oct 31 13:31:30.060276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.060247    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e49afd3-26bd-493f-a807-ccd12c325d74-kube-api-access-7jfs4\" (UniqueName: \"kubernetes.io/projected/7e49afd3-26bd-493f-a807-ccd12c325d74-kube-api-access-7jfs4\") pod \"7e49afd3-26bd-493f-a807-ccd12c325d74\" (UID: \"7e49afd3-26bd-493f-a807-ccd12c325d74\") "
Oct 31 13:31:30.060427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.060376    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e49afd3-26bd-493f-a807-ccd12c325d74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e49afd3-26bd-493f-a807-ccd12c325d74-plugins-config-volume\") pod \"7e49afd3-26bd-493f-a807-ccd12c325d74\" (UID: \"7e49afd3-26bd-493f-a807-ccd12c325d74\") "
Oct 31 13:31:30.060733 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.060707    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e49afd3-26bd-493f-a807-ccd12c325d74-plugins-config-volume" pod "7e49afd3-26bd-493f-a807-ccd12c325d74" (UID: "7e49afd3-26bd-493f-a807-ccd12c325d74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:31:30.061217 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.061197    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e49afd3-26bd-493f-a807-ccd12c325d74-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:30.062532 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.062514    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e49afd3-26bd-493f-a807-ccd12c325d74-kube-api-access-7jfs4" pod "7e49afd3-26bd-493f-a807-ccd12c325d74" (UID: "7e49afd3-26bd-493f-a807-ccd12c325d74"). InnerVolumeSpecName "kube-api-access-7jfs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:31:30.146895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.146850    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs" podUID="8e4b37cc-b081-4b6b-a71b-21b6cfc5766f" containerName="payload-processing" containerID="cri-o://cd3fa0b04ef4a41c661bac2eebe6bf29a9709e79eb09673ae35ad0641c8d63a5" gracePeriod=25
Oct 31 13:31:30.162861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.162836    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jfs4\" (UniqueName: \"kubernetes.io/projected/7e49afd3-26bd-493f-a807-ccd12c325d74-kube-api-access-7jfs4\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:30.407218 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.407179    4428 generic.go:356] "Generic (PLEG): container finished" podID="8e4b37cc-b081-4b6b-a71b-21b6cfc5766f" containerID="cd3fa0b04ef4a41c661bac2eebe6bf29a9709e79eb09673ae35ad0641c8d63a5" exitCode=0
Oct 31 13:31:30.408588 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.407619    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs" event={"ID":"8e4b37cc-b081-4b6b-a71b-21b6cfc5766f","Type":"ContainerDied","Data":"cd3fa0b04ef4a41c661bac2eebe6bf29a9709e79eb09673ae35ad0641c8d63a5"}
Oct 31 13:31:30.411705 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.411676    4428 generic.go:356] "Generic (PLEG): container finished" podID="a5ef0238-0e04-42d5-93b2-625f05dbaeb2" containerID="a7bf51823cc388bb4ea437997fe510787c89469904dc39820b98d3baaaf45aac" exitCode=0
Oct 31 13:31:30.412064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.412041    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829" event={"ID":"a5ef0238-0e04-42d5-93b2-625f05dbaeb2","Type":"ContainerDied","Data":"a7bf51823cc388bb4ea437997fe510787c89469904dc39820b98d3baaaf45aac"}
Oct 31 13:31:30.412214 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.412199    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829" event={"ID":"a5ef0238-0e04-42d5-93b2-625f05dbaeb2","Type":"ContainerDied","Data":"6a42bb62c6a05ef65e2d1ea6ebe5732f7b0233e12b08bdc4b69c3d5cd58d3a2f"}
Oct 31 13:31:30.412339 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.412327    4428 scope.go:122] "RemoveContainer" containerID="a7bf51823cc388bb4ea437997fe510787c89469904dc39820b98d3baaaf45aac"
Oct 31 13:31:30.412618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.412603    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"
Oct 31 13:31:30.436332 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.436296    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:30.442811 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.442779    4428 generic.go:356] "Generic (PLEG): container finished" podID="7e49afd3-26bd-493f-a807-ccd12c325d74" containerID="472047b68ae71da26ca3363fbcae4644c270047302a782e1ad1632d86f1588e6" exitCode=0
Oct 31 13:31:30.442987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.442828    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv" event={"ID":"7e49afd3-26bd-493f-a807-ccd12c325d74","Type":"ContainerDied","Data":"472047b68ae71da26ca3363fbcae4644c270047302a782e1ad1632d86f1588e6"}
Oct 31 13:31:30.442987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.442906    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv" event={"ID":"7e49afd3-26bd-493f-a807-ccd12c325d74","Type":"ContainerDied","Data":"79ad9df6592b6c272865196e87057c33e24c372df6d8fef4f9386f1766998cfa"}
Oct 31 13:31:30.443119 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.443014    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"
Oct 31 13:31:30.585137 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.579058    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"]
Oct 31 13:31:30.588621 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.588563    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c7b990ba-6d4457fc6f-ww829"]
Oct 31 13:31:30.597592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.597572    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"]
Oct 31 13:31:30.600424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.600408    4428 scope.go:122] "RemoveContainer" containerID="a7bf51823cc388bb4ea437997fe510787c89469904dc39820b98d3baaaf45aac"
Oct 31 13:31:30.601060 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:31:30.601039    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7bf51823cc388bb4ea437997fe510787c89469904dc39820b98d3baaaf45aac\": container with ID starting with a7bf51823cc388bb4ea437997fe510787c89469904dc39820b98d3baaaf45aac not found: ID does not exist" containerID="a7bf51823cc388bb4ea437997fe510787c89469904dc39820b98d3baaaf45aac"
Oct 31 13:31:30.601163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.601074    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7bf51823cc388bb4ea437997fe510787c89469904dc39820b98d3baaaf45aac"} err="failed to get container status \"a7bf51823cc388bb4ea437997fe510787c89469904dc39820b98d3baaaf45aac\": rpc error: code = NotFound desc = could not find container \"a7bf51823cc388bb4ea437997fe510787c89469904dc39820b98d3baaaf45aac\": container with ID starting with a7bf51823cc388bb4ea437997fe510787c89469904dc39820b98d3baaaf45aac not found: ID does not exist"
Oct 31 13:31:30.601163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.601097    4428 scope.go:122] "RemoveContainer" containerID="472047b68ae71da26ca3363fbcae4644c270047302a782e1ad1632d86f1588e6"
Oct 31 13:31:30.605061 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.605040    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c7b990ba-57bfc7df59-vlnxv"]
Oct 31 13:31:30.617771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.617754    4428 scope.go:122] "RemoveContainer" containerID="472047b68ae71da26ca3363fbcae4644c270047302a782e1ad1632d86f1588e6"
Oct 31 13:31:30.618008 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:31:30.617989    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"472047b68ae71da26ca3363fbcae4644c270047302a782e1ad1632d86f1588e6\": container with ID starting with 472047b68ae71da26ca3363fbcae4644c270047302a782e1ad1632d86f1588e6 not found: ID does not exist" containerID="472047b68ae71da26ca3363fbcae4644c270047302a782e1ad1632d86f1588e6"
Oct 31 13:31:30.618098 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.618014    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"472047b68ae71da26ca3363fbcae4644c270047302a782e1ad1632d86f1588e6"} err="failed to get container status \"472047b68ae71da26ca3363fbcae4644c270047302a782e1ad1632d86f1588e6\": rpc error: code = NotFound desc = could not find container \"472047b68ae71da26ca3363fbcae4644c270047302a782e1ad1632d86f1588e6\": container with ID starting with 472047b68ae71da26ca3363fbcae4644c270047302a782e1ad1632d86f1588e6 not found: ID does not exist"
Oct 31 13:31:30.623593 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.623574    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"
Oct 31 13:31:30.677915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.677829    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4b37cc-b081-4b6b-a71b-21b6cfc5766f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4b37cc-b081-4b6b-a71b-21b6cfc5766f-plugins-config-volume\") pod \"8e4b37cc-b081-4b6b-a71b-21b6cfc5766f\" (UID: \"8e4b37cc-b081-4b6b-a71b-21b6cfc5766f\") "
Oct 31 13:31:30.678187 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.678165    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4b37cc-b081-4b6b-a71b-21b6cfc5766f-kube-api-access-hxn8s\" (UniqueName: \"kubernetes.io/projected/8e4b37cc-b081-4b6b-a71b-21b6cfc5766f-kube-api-access-hxn8s\") pod \"8e4b37cc-b081-4b6b-a71b-21b6cfc5766f\" (UID: \"8e4b37cc-b081-4b6b-a71b-21b6cfc5766f\") "
Oct 31 13:31:30.678321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.678295    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4b37cc-b081-4b6b-a71b-21b6cfc5766f-plugins-config-volume" pod "8e4b37cc-b081-4b6b-a71b-21b6cfc5766f" (UID: "8e4b37cc-b081-4b6b-a71b-21b6cfc5766f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:31:30.678948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.678921    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4b37cc-b081-4b6b-a71b-21b6cfc5766f-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:30.681327 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.681288    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4b37cc-b081-4b6b-a71b-21b6cfc5766f-kube-api-access-hxn8s" pod "8e4b37cc-b081-4b6b-a71b-21b6cfc5766f" (UID: "8e4b37cc-b081-4b6b-a71b-21b6cfc5766f"). InnerVolumeSpecName "kube-api-access-hxn8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:31:30.759012 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.758986    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e49afd3-26bd-493f-a807-ccd12c325d74" path="/var/lib/kubelet/pods/7e49afd3-26bd-493f-a807-ccd12c325d74/volumes"
Oct 31 13:31:30.759567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.759550    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ef0238-0e04-42d5-93b2-625f05dbaeb2" path="/var/lib/kubelet/pods/a5ef0238-0e04-42d5-93b2-625f05dbaeb2/volumes"
Oct 31 13:31:30.780625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:30.780599    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxn8s\" (UniqueName: \"kubernetes.io/projected/8e4b37cc-b081-4b6b-a71b-21b6cfc5766f-kube-api-access-hxn8s\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:31.491936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:31.491519    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs" event={"ID":"8e4b37cc-b081-4b6b-a71b-21b6cfc5766f","Type":"ContainerDied","Data":"5763bc6c6de423f21c4d1bfabe97a4d5930a07c7f42f95884ea69bc4f2f20403"}
Oct 31 13:31:31.491936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:31.491595    4428 scope.go:122] "RemoveContainer" containerID="cd3fa0b04ef4a41c661bac2eebe6bf29a9709e79eb09673ae35ad0641c8d63a5"
Oct 31 13:31:31.491936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:31.491712    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"
Oct 31 13:31:31.520426 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:31.520393    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"]
Oct 31 13:31:31.526178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:31.526158    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8439b78a-7c455dfdc5-z24rs"]
Oct 31 13:31:31.660337 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:31.660302    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:32.660891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:32.652264    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:32.761833 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:32.761795    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4b37cc-b081-4b6b-a71b-21b6cfc5766f" path="/var/lib/kubelet/pods/8e4b37cc-b081-4b6b-a71b-21b6cfc5766f/volumes"
Oct 31 13:31:33.459950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:33.459889    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:33.475462 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:33.475430    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p" podUID="8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa" containerName="payload-processing" containerID="cri-o://8feaab8151f2a2c4d9fb1c92fd7f395c2ac44883e331cca2711397482a3b6322" gracePeriod=25
Oct 31 13:31:33.712946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:33.703431    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:34.069844 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.068264    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:34.092899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.085922    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"]
Oct 31 13:31:34.112945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.112418    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8439b78a-c59kn"]
Oct 31 13:31:34.234781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.234335    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:34.375742 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.375719    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"
Oct 31 13:31:34.411911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.411823    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:34.416282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.416256    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa-kube-api-access-5nllv\" (UniqueName: \"kubernetes.io/projected/8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa-kube-api-access-5nllv\") pod \"8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa\" (UID: \"8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa\") "
Oct 31 13:31:34.416435 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.416420    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa-plugins-config-volume\") pod \"8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa\" (UID: \"8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa\") "
Oct 31 13:31:34.416781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.416758    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa-plugins-config-volume" pod "8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa" (UID: "8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:31:34.417051 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.417033    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:34.420730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.418744    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa-kube-api-access-5nllv" pod "8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa" (UID: "8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa"). InnerVolumeSpecName "kube-api-access-5nllv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:31:34.558739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.558700    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nllv\" (UniqueName: \"kubernetes.io/projected/8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa-kube-api-access-5nllv\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:34.842451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.836415    4428 generic.go:356] "Generic (PLEG): container finished" podID="8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa" containerID="8feaab8151f2a2c4d9fb1c92fd7f395c2ac44883e331cca2711397482a3b6322" exitCode=0
Oct 31 13:31:34.842451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.836569    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"
Oct 31 13:31:34.851404 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.851368    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p" event={"ID":"8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa","Type":"ContainerDied","Data":"8feaab8151f2a2c4d9fb1c92fd7f395c2ac44883e331cca2711397482a3b6322"}
Oct 31 13:31:34.853781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.853306    4428 scope.go:122] "RemoveContainer" containerID="8feaab8151f2a2c4d9fb1c92fd7f395c2ac44883e331cca2711397482a3b6322"
Oct 31 13:31:34.870914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.870837    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c19b695c-87c3-41b8-bb49-4a671676a4e4" path="/var/lib/kubelet/pods/c19b695c-87c3-41b8-bb49-4a671676a4e4/volumes"
Oct 31 13:31:34.871634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.871603    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p" event={"ID":"8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa","Type":"ContainerDied","Data":"d1328130802ff1dbe4f2b96c2e4dbe7a10f656f13682ebeb9ea57dd6f989a1ab"}
Oct 31 13:31:34.874848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.874833    4428 scope.go:122] "RemoveContainer" containerID="8feaab8151f2a2c4d9fb1c92fd7f395c2ac44883e331cca2711397482a3b6322"
Oct 31 13:31:34.875236 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:31:34.875190    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8feaab8151f2a2c4d9fb1c92fd7f395c2ac44883e331cca2711397482a3b6322\": container with ID starting with 8feaab8151f2a2c4d9fb1c92fd7f395c2ac44883e331cca2711397482a3b6322 not found: ID does not exist" containerID="8feaab8151f2a2c4d9fb1c92fd7f395c2ac44883e331cca2711397482a3b6322"
Oct 31 13:31:34.875236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.875220    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8feaab8151f2a2c4d9fb1c92fd7f395c2ac44883e331cca2711397482a3b6322"} err="failed to get container status \"8feaab8151f2a2c4d9fb1c92fd7f395c2ac44883e331cca2711397482a3b6322\": rpc error: code = NotFound desc = could not find container \"8feaab8151f2a2c4d9fb1c92fd7f395c2ac44883e331cca2711397482a3b6322\": container with ID starting with 8feaab8151f2a2c4d9fb1c92fd7f395c2ac44883e331cca2711397482a3b6322 not found: ID does not exist"
Oct 31 13:31:34.891649 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.891625    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"]
Oct 31 13:31:34.899618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:34.899589    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc1bbe2b-55f58c5867-7ns2p"]
Oct 31 13:31:35.093198 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.093114    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:35.301354 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.301304    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:35.796995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.794095    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"]
Oct 31 13:31:35.796995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.794424    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:35.802962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.802127    4428 cpu_manager.go:414] "containerMap: removing container" podUID="c19b695c-87c3-41b8-bb49-4a671676a4e4" containerName="revoke-keys"
Oct 31 13:31:35.802962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.802156    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c19b695c-87c3-41b8-bb49-4a671676a4e4" containerName="revoke-keys"
Oct 31 13:31:35.802962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.802196    4428 cpu_manager.go:414] "containerMap: removing container" podUID="8e4b37cc-b081-4b6b-a71b-21b6cfc5766f" containerName="payload-processing"
Oct 31 13:31:35.802962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.802204    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4b37cc-b081-4b6b-a71b-21b6cfc5766f" containerName="payload-processing"
Oct 31 13:31:35.802962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.802219    4428 cpu_manager.go:414] "containerMap: removing container" podUID="a5ef0238-0e04-42d5-93b2-625f05dbaeb2" containerName="payload-processing"
Oct 31 13:31:35.802962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.802226    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ef0238-0e04-42d5-93b2-625f05dbaeb2" containerName="payload-processing"
Oct 31 13:31:35.802962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.802252    4428 cpu_manager.go:414] "containerMap: removing container" podUID="8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa" containerName="payload-processing"
Oct 31 13:31:35.802962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.802259    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa" containerName="payload-processing"
Oct 31 13:31:35.802962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.802287    4428 cpu_manager.go:414] "containerMap: removing container" podUID="7e49afd3-26bd-493f-a807-ccd12c325d74" containerName="payload-pre-processing"
Oct 31 13:31:35.802962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.802294    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e49afd3-26bd-493f-a807-ccd12c325d74" containerName="payload-pre-processing"
Oct 31 13:31:35.802962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.802520    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4b37cc-b081-4b6b-a71b-21b6cfc5766f" containerName="payload-processing"
Oct 31 13:31:35.802962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.802540    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ef0238-0e04-42d5-93b2-625f05dbaeb2" containerName="payload-processing"
Oct 31 13:31:35.802962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.802553    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa" containerName="payload-processing"
Oct 31 13:31:35.802962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.802565    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e49afd3-26bd-493f-a807-ccd12c325d74" containerName="payload-pre-processing"
Oct 31 13:31:35.802962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.802580    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c19b695c-87c3-41b8-bb49-4a671676a4e4" containerName="revoke-keys"
Oct 31 13:31:35.825800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.823111    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"]
Oct 31 13:31:35.825800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.823233    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.887293 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.887259    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.887635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.887380    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.887635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.887407    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8s2s\" (UniqueName: \"kubernetes.io/projected/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-kube-api-access-f8s2s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.887635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.887477    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.887635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.887620    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.887813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.887723    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.887813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.887767    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.989206 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.989171    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.989378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.989246    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.989378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.989294    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.989378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.989372    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.989562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.989406    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8s2s\" (UniqueName: \"kubernetes.io/projected/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-kube-api-access-f8s2s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.989562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.989437    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.989562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.989546    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.990136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.990114    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.991805 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.991777    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.991947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.991858    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.991995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.991948    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:35.992253 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:35.992237    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:36.002652 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:36.002626    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:36.002843 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:36.002825    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8s2s\" (UniqueName: \"kubernetes.io/projected/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-kube-api-access-f8s2s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wb6gv\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:36.186140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:36.186062    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:36.493189 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:36.493159    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"]
Oct 31 13:31:36.498836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:36.498792    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:31:36.499040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:36.498912    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:31:36.583960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:36.583928    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:36.872571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:36.872493    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa" path="/var/lib/kubelet/pods/8d02f56a-71e6-4ca8-b1e2-e24b7b21f5fa/volumes"
Oct 31 13:31:36.908895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:36.903060    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv" event={"ID":"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2","Type":"ContainerStarted","Data":"ad97f22244d8b644cdb74aa025e760f718f0625af825cbee5003f97d808b75aa"}
Oct 31 13:31:36.908895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:36.903121    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv" event={"ID":"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2","Type":"ContainerStarted","Data":"96853bdb9097a411ea49ff7ad8a61122437b338c8fb3ef266b3de714a3f2d64a"}
Oct 31 13:31:36.908895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:36.904493    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:36.993598 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:36.993566    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:37.204016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:37.196405    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:37.499905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:37.495357    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:37.778544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:37.778465    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:38.273892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:38.269584    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv" podStartSLOduration=3.269557131 podStartE2EDuration="3.269557131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:36.934342501 +0000 UTC m=+1694.332691127" watchObservedRunningTime="2026-10-31 13:31:38.269557131 +0000 UTC m=+1695.667905757"
Oct 31 13:31:38.360962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:38.339615    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"]
Oct 31 13:31:38.360962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:38.340123    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw" podUID="b454c7dc-1585-4c9a-9d89-e47285739d60" containerName="maas-api" containerID="cri-o://e0463ddeca19d8a4d331e3eca6f640d209823a6edb6f5434cf8a04331f6d099b" gracePeriod=30
Oct 31 13:31:38.380946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:38.376679    4428 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wb6gv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:31:38.380946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:38.376743    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv" podUID="e899d7ff-9dd2-48d8-b537-f03e45b6d1c2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:31:38.401622 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:38.400549    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:38.570990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:38.570896    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:38.691928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:38.691861    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"]
Oct 31 13:31:38.776893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:38.773784    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:39.015898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:39.012426    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:39.096172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:39.096142    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c7b990ba-stwxt"]
Oct 31 13:31:39.612078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:39.612041    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:39.976960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:39.961109    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:40.305134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.291222    4428 generic.go:356] "Generic (PLEG): container finished" podID="b454c7dc-1585-4c9a-9d89-e47285739d60" containerID="e0463ddeca19d8a4d331e3eca6f640d209823a6edb6f5434cf8a04331f6d099b" exitCode=0
Oct 31 13:31:40.326249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.300304    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw" event={"ID":"b454c7dc-1585-4c9a-9d89-e47285739d60","Type":"ContainerDied","Data":"e0463ddeca19d8a4d331e3eca6f640d209823a6edb6f5434cf8a04331f6d099b"}
Oct 31 13:31:40.326249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.300504    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:40.344282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.344254    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:31:40.546280 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.545692    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:40.666161 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.666075    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:31:40.778993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.778960    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b454c7dc-1585-4c9a-9d89-e47285739d60-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b454c7dc-1585-4c9a-9d89-e47285739d60-metrics-certs\") pod \"b454c7dc-1585-4c9a-9d89-e47285739d60\" (UID: \"b454c7dc-1585-4c9a-9d89-e47285739d60\") "
Oct 31 13:31:40.779374 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.779355    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b454c7dc-1585-4c9a-9d89-e47285739d60-kube-api-access-rt8jj\" (UniqueName: \"kubernetes.io/projected/b454c7dc-1585-4c9a-9d89-e47285739d60-kube-api-access-rt8jj\") pod \"b454c7dc-1585-4c9a-9d89-e47285739d60\" (UID: \"b454c7dc-1585-4c9a-9d89-e47285739d60\") "
Oct 31 13:31:40.779591 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.779576    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b454c7dc-1585-4c9a-9d89-e47285739d60-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b454c7dc-1585-4c9a-9d89-e47285739d60-openshift-service-ca\") pod \"b454c7dc-1585-4c9a-9d89-e47285739d60\" (UID: \"b454c7dc-1585-4c9a-9d89-e47285739d60\") "
Oct 31 13:31:40.779848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.779831    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b454c7dc-1585-4c9a-9d89-e47285739d60-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b454c7dc-1585-4c9a-9d89-e47285739d60-maas-api-tls\") pod \"b454c7dc-1585-4c9a-9d89-e47285739d60\" (UID: \"b454c7dc-1585-4c9a-9d89-e47285739d60\") "
Oct 31 13:31:40.793949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.793369    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b454c7dc-1585-4c9a-9d89-e47285739d60-openshift-service-ca" pod "b454c7dc-1585-4c9a-9d89-e47285739d60" (UID: "b454c7dc-1585-4c9a-9d89-e47285739d60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:31:40.809956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.809903    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b454c7dc-1585-4c9a-9d89-e47285739d60-kube-api-access-rt8jj" pod "b454c7dc-1585-4c9a-9d89-e47285739d60" (UID: "b454c7dc-1585-4c9a-9d89-e47285739d60"). InnerVolumeSpecName "kube-api-access-rt8jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:31:40.815504 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.815456    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b454c7dc-1585-4c9a-9d89-e47285739d60-metrics-certs" pod "b454c7dc-1585-4c9a-9d89-e47285739d60" (UID: "b454c7dc-1585-4c9a-9d89-e47285739d60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:31:40.816476 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.816450    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b454c7dc-1585-4c9a-9d89-e47285739d60-maas-api-tls" pod "b454c7dc-1585-4c9a-9d89-e47285739d60" (UID: "b454c7dc-1585-4c9a-9d89-e47285739d60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:31:40.827052 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.827020    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2" path="/var/lib/kubelet/pods/711e30cb-0fbe-41bc-9cd7-a1e9c1137eb2/volumes"
Oct 31 13:31:40.884910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.882697    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt8jj\" (UniqueName: \"kubernetes.io/projected/b454c7dc-1585-4c9a-9d89-e47285739d60-kube-api-access-rt8jj\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:40.884910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.882733    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b454c7dc-1585-4c9a-9d89-e47285739d60-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:40.884910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.882750    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b454c7dc-1585-4c9a-9d89-e47285739d60-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:40.884910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:40.882764    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b454c7dc-1585-4c9a-9d89-e47285739d60-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:41.316656 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:41.316628    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"
Oct 31 13:31:41.316656 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:41.316650    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw" event={"ID":"b454c7dc-1585-4c9a-9d89-e47285739d60","Type":"ContainerDied","Data":"7fb7f1894f95ad8f2ad40ad3d208c3feef7cc84d5f078e59ff3df019b25552e1"}
Oct 31 13:31:41.316921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:41.316722    4428 scope.go:122] "RemoveContainer" containerID="e0463ddeca19d8a4d331e3eca6f640d209823a6edb6f5434cf8a04331f6d099b"
Oct 31 13:31:41.348273 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:41.348243    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"]
Oct 31 13:31:41.355514 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:41.355493    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7b990ba-b77476f8c-ftwsw"]
Oct 31 13:31:41.997473 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:41.997436    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:42.757852 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:42.757819    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b454c7dc-1585-4c9a-9d89-e47285739d60" path="/var/lib/kubelet/pods/b454c7dc-1585-4c9a-9d89-e47285739d60/volumes"
Oct 31 13:31:43.062951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:43.062376    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:44.029908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:44.029796    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:44.556082 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:44.556022    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:44.781657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:44.781625    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:45.090561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:45.090529    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:45.274571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:45.274531    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:45.797920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:45.797844    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:45.959339 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:45.959305    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:46.142639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.132014    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:46.486568 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.486535    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"]
Oct 31 13:31:46.487679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.487657    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b454c7dc-1585-4c9a-9d89-e47285739d60" containerName="maas-api"
Oct 31 13:31:46.487850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.487837    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b454c7dc-1585-4c9a-9d89-e47285739d60" containerName="maas-api"
Oct 31 13:31:46.488205 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.488190    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b454c7dc-1585-4c9a-9d89-e47285739d60" containerName="maas-api"
Oct 31 13:31:46.494891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.494852    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"
Oct 31 13:31:46.497788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.497767    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"]
Oct 31 13:31:46.500537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.500509    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:31:46.500667 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.500646    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:31:46.655987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.655953    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:46.679197 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.679162    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qhd4\" (UniqueName: \"kubernetes.io/projected/b5b7ad21-f603-492f-8096-89922a09b4ce-kube-api-access-4qhd4\") pod \"maas-api-revoke-keys-e2e-ait-dff70365-mm4vz\" (UID: \"b5b7ad21-f603-492f-8096-89922a09b4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"
Oct 31 13:31:46.679340 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.679323    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b7ad21-f603-492f-8096-89922a09b4ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dff70365-mm4vz\" (UID: \"b5b7ad21-f603-492f-8096-89922a09b4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"
Oct 31 13:31:46.679736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.679710    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b7ad21-f603-492f-8096-89922a09b4ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dff70365-mm4vz\" (UID: \"b5b7ad21-f603-492f-8096-89922a09b4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"
Oct 31 13:31:46.781071 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.780994    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b7ad21-f603-492f-8096-89922a09b4ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dff70365-mm4vz\" (UID: \"b5b7ad21-f603-492f-8096-89922a09b4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"
Oct 31 13:31:46.781071 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.781043    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qhd4\" (UniqueName: \"kubernetes.io/projected/b5b7ad21-f603-492f-8096-89922a09b4ce-kube-api-access-4qhd4\") pod \"maas-api-revoke-keys-e2e-ait-dff70365-mm4vz\" (UID: \"b5b7ad21-f603-492f-8096-89922a09b4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"
Oct 31 13:31:46.781286 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.781108    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b7ad21-f603-492f-8096-89922a09b4ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dff70365-mm4vz\" (UID: \"b5b7ad21-f603-492f-8096-89922a09b4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"
Oct 31 13:31:46.781653 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.781629    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b7ad21-f603-492f-8096-89922a09b4ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dff70365-mm4vz\" (UID: \"b5b7ad21-f603-492f-8096-89922a09b4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"
Oct 31 13:31:46.783741 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.783718    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b7ad21-f603-492f-8096-89922a09b4ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dff70365-mm4vz\" (UID: \"b5b7ad21-f603-492f-8096-89922a09b4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"
Oct 31 13:31:46.795604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.795578    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qhd4\" (UniqueName: \"kubernetes.io/projected/b5b7ad21-f603-492f-8096-89922a09b4ce-kube-api-access-4qhd4\") pod \"maas-api-revoke-keys-e2e-ait-dff70365-mm4vz\" (UID: \"b5b7ad21-f603-492f-8096-89922a09b4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"
Oct 31 13:31:46.832904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.828208    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"
Oct 31 13:31:46.848367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:46.848263    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:47.295998 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:31:47.295962    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b7ad21_f603_492f_8096_89922a09b4ce.slice/crio-a6e95c8645b2221f14bc3cd531d0eb276ba7a18f6ada0232050e8c6f8750175f WatchSource:0}: Error finding container a6e95c8645b2221f14bc3cd531d0eb276ba7a18f6ada0232050e8c6f8750175f: Status 404 returned error can't find the container with id a6e95c8645b2221f14bc3cd531d0eb276ba7a18f6ada0232050e8c6f8750175f
Oct 31 13:31:47.297465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:47.297440    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"]
Oct 31 13:31:47.409895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:47.408991    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"]
Oct 31 13:31:47.512272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:47.511693    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:47.514544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:47.513809    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz" event={"ID":"b5b7ad21-f603-492f-8096-89922a09b4ce","Type":"ContainerStarted","Data":"a6e95c8645b2221f14bc3cd531d0eb276ba7a18f6ada0232050e8c6f8750175f"}
Oct 31 13:31:47.671051 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:47.670975    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:48.107893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:48.094261    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"]
Oct 31 13:31:48.107893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:48.103902    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b7aeae-xgx25"]
Oct 31 13:31:48.199730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:48.199688    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:48.524207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:48.524169    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz" event={"ID":"b5b7ad21-f603-492f-8096-89922a09b4ce","Type":"ContainerStarted","Data":"e6789be8ed37c1bca537cf625f62d723e6626ce5eeb8da53944a94bb5fbcc288"}
Oct 31 13:31:48.554095 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:48.554037    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz" podStartSLOduration=2.554022888 podStartE2EDuration="2.554022888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:48.549590159 +0000 UTC m=+1705.947938784" watchObservedRunningTime="2026-10-31 13:31:48.554022888 +0000 UTC m=+1705.952371535"
Oct 31 13:31:48.741297 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:31:48.741258    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:31:48.756599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:48.756570    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="053fc722-bf18-4cc5-a5d2-884b8893a5a8" path="/var/lib/kubelet/pods/053fc722-bf18-4cc5-a5d2-884b8893a5a8/volumes"
Oct 31 13:31:49.126893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:49.126093    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:49.534454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:49.534417    4428 generic.go:356] "Generic (PLEG): container finished" podID="b5b7ad21-f603-492f-8096-89922a09b4ce" containerID="e6789be8ed37c1bca537cf625f62d723e6626ce5eeb8da53944a94bb5fbcc288" exitCode=0
Oct 31 13:31:49.534667 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:49.534496    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz" event={"ID":"b5b7ad21-f603-492f-8096-89922a09b4ce","Type":"ContainerDied","Data":"e6789be8ed37c1bca537cf625f62d723e6626ce5eeb8da53944a94bb5fbcc288"}
Oct 31 13:31:49.991004 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:49.990970    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:50.725322 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:50.725300    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"
Oct 31 13:31:50.853835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:50.853752    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b7ad21-f603-492f-8096-89922a09b4ce-kube-api-access-4qhd4\" (UniqueName: \"kubernetes.io/projected/b5b7ad21-f603-492f-8096-89922a09b4ce-kube-api-access-4qhd4\") pod \"b5b7ad21-f603-492f-8096-89922a09b4ce\" (UID: \"b5b7ad21-f603-492f-8096-89922a09b4ce\") "
Oct 31 13:31:50.853835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:50.853825    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5b7ad21-f603-492f-8096-89922a09b4ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b7ad21-f603-492f-8096-89922a09b4ce-maas-api-xks-ca\") pod \"b5b7ad21-f603-492f-8096-89922a09b4ce\" (UID: \"b5b7ad21-f603-492f-8096-89922a09b4ce\") "
Oct 31 13:31:50.854111 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:50.853859    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b7ad21-f603-492f-8096-89922a09b4ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b7ad21-f603-492f-8096-89922a09b4ce-maas-api-openshift-ca\") pod \"b5b7ad21-f603-492f-8096-89922a09b4ce\" (UID: \"b5b7ad21-f603-492f-8096-89922a09b4ce\") "
Oct 31 13:31:50.856153 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:50.856125    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b7ad21-f603-492f-8096-89922a09b4ce-kube-api-access-4qhd4" pod "b5b7ad21-f603-492f-8096-89922a09b4ce" (UID: "b5b7ad21-f603-492f-8096-89922a09b4ce"). InnerVolumeSpecName "kube-api-access-4qhd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:31:50.857203 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:50.857179    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b7ad21-f603-492f-8096-89922a09b4ce-maas-api-openshift-ca" pod "b5b7ad21-f603-492f-8096-89922a09b4ce" (UID: "b5b7ad21-f603-492f-8096-89922a09b4ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:31:50.858859 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:50.858834    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b7ad21-f603-492f-8096-89922a09b4ce-maas-api-xks-ca" pod "b5b7ad21-f603-492f-8096-89922a09b4ce" (UID: "b5b7ad21-f603-492f-8096-89922a09b4ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:31:50.977894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:50.966763    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qhd4\" (UniqueName: \"kubernetes.io/projected/b5b7ad21-f603-492f-8096-89922a09b4ce-kube-api-access-4qhd4\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:50.977894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:50.966800    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b7ad21-f603-492f-8096-89922a09b4ce-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:50.977894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:50.966815    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b7ad21-f603-492f-8096-89922a09b4ce-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:51.002540 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:51.002509    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:51.552332 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:51.552303    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"
Oct 31 13:31:51.552502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:51.552348    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz" event={"ID":"b5b7ad21-f603-492f-8096-89922a09b4ce","Type":"ContainerDied","Data":"a6e95c8645b2221f14bc3cd531d0eb276ba7a18f6ada0232050e8c6f8750175f"}
Oct 31 13:31:51.552502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:51.552414    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6e95c8645b2221f14bc3cd531d0eb276ba7a18f6ada0232050e8c6f8750175f"
Oct 31 13:31:51.839217 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:51.839121    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:52.419108 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:52.419071    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v" podUID="1004aeb3-63d7-48e5-ac93-59eea0a4c66a" containerName="payload-processing" containerID="cri-o://64ac1ca52d47d70e1cc0b7a36bb90a28b260fcb1724e0900e19e54bd26a3fbd5" gracePeriod=25
Oct 31 13:31:52.562948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:52.562789    4428 generic.go:356] "Generic (PLEG): container finished" podID="1004aeb3-63d7-48e5-ac93-59eea0a4c66a" containerID="64ac1ca52d47d70e1cc0b7a36bb90a28b260fcb1724e0900e19e54bd26a3fbd5" exitCode=0
Oct 31 13:31:52.562948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:52.562910    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v" event={"ID":"1004aeb3-63d7-48e5-ac93-59eea0a4c66a","Type":"ContainerDied","Data":"64ac1ca52d47d70e1cc0b7a36bb90a28b260fcb1724e0900e19e54bd26a3fbd5"}
Oct 31 13:31:52.624252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:52.624105    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:52.855796 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:52.855772    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"
Oct 31 13:31:52.901949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:52.901915    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1004aeb3-63d7-48e5-ac93-59eea0a4c66a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1004aeb3-63d7-48e5-ac93-59eea0a4c66a-plugins-config-volume\") pod \"1004aeb3-63d7-48e5-ac93-59eea0a4c66a\" (UID: \"1004aeb3-63d7-48e5-ac93-59eea0a4c66a\") "
Oct 31 13:31:52.902125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:52.902002    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1004aeb3-63d7-48e5-ac93-59eea0a4c66a-kube-api-access-gps56\" (UniqueName: \"kubernetes.io/projected/1004aeb3-63d7-48e5-ac93-59eea0a4c66a-kube-api-access-gps56\") pod \"1004aeb3-63d7-48e5-ac93-59eea0a4c66a\" (UID: \"1004aeb3-63d7-48e5-ac93-59eea0a4c66a\") "
Oct 31 13:31:52.902248 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:52.902223    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1004aeb3-63d7-48e5-ac93-59eea0a4c66a-plugins-config-volume" pod "1004aeb3-63d7-48e5-ac93-59eea0a4c66a" (UID: "1004aeb3-63d7-48e5-ac93-59eea0a4c66a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:31:52.902804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:52.902775    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1004aeb3-63d7-48e5-ac93-59eea0a4c66a-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:52.904374 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:52.904349    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1004aeb3-63d7-48e5-ac93-59eea0a4c66a-kube-api-access-gps56" pod "1004aeb3-63d7-48e5-ac93-59eea0a4c66a" (UID: "1004aeb3-63d7-48e5-ac93-59eea0a4c66a"). InnerVolumeSpecName "kube-api-access-gps56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:31:53.004239 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:53.004208    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gps56\" (UniqueName: \"kubernetes.io/projected/1004aeb3-63d7-48e5-ac93-59eea0a4c66a-kube-api-access-gps56\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:53.516895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:53.514967    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:53.597563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:53.597517    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v" event={"ID":"1004aeb3-63d7-48e5-ac93-59eea0a4c66a","Type":"ContainerDied","Data":"493c21eb622a4391472925eced646a9a82829200b87e99ad276165cd2dd6199d"}
Oct 31 13:31:53.597759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:53.597612    4428 scope.go:122] "RemoveContainer" containerID="64ac1ca52d47d70e1cc0b7a36bb90a28b260fcb1724e0900e19e54bd26a3fbd5"
Oct 31 13:31:53.597759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:53.597537    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"
Oct 31 13:31:53.629213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:53.629191    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"]
Oct 31 13:31:53.634366 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:53.634342    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a66ae960-f4d46755-jxq5v"]
Oct 31 13:31:54.348613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:54.348577    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:54.555702 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:54.555666    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:54.756735 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:54.756700    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1004aeb3-63d7-48e5-ac93-59eea0a4c66a" path="/var/lib/kubelet/pods/1004aeb3-63d7-48e5-ac93-59eea0a4c66a/volumes"
Oct 31 13:31:55.338275 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:55.338239    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:55.530623 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:55.530598    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:55.735858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:55.735821    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:56.448264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:56.448232    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:56.900897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:56.861525    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:57.221926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:57.220399    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"]
Oct 31 13:31:57.230296 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:57.230262    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:57.233635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:57.233601    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b2971e-wvdvz"]
Oct 31 13:31:57.636087 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:57.635986    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"]
Oct 31 13:31:57.636604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:57.636527    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb" podUID="e799f52e-4fcc-47be-b42c-d43976f7660f" containerName="maas-api" containerID="cri-o://4b81cf0cd81b0f911ab21567935b50ca649be412b4e9e14fcf506e3ee56227fa" gracePeriod=30
Oct 31 13:31:57.671908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:57.668700    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:58.042899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:58.025763    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:58.690248 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:58.664733    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:58.799286 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:58.735759    4428 generic.go:356] "Generic (PLEG): container finished" podID="e799f52e-4fcc-47be-b42c-d43976f7660f" containerID="4b81cf0cd81b0f911ab21567935b50ca649be412b4e9e14fcf506e3ee56227fa" exitCode=0
Oct 31 13:31:58.799286 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:58.735991    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb" event={"ID":"e799f52e-4fcc-47be-b42c-d43976f7660f","Type":"ContainerDied","Data":"4b81cf0cd81b0f911ab21567935b50ca649be412b4e9e14fcf506e3ee56227fa"}
Oct 31 13:31:59.020645 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.020155    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:59.024989 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.024943    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c06662-996a-4bf1-a02b-673daa51c549" path="/var/lib/kubelet/pods/f1c06662-996a-4bf1-a02b-673daa51c549/volumes"
Oct 31 13:31:59.028807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.028785    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:31:59.097590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.097552    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e799f52e-4fcc-47be-b42c-d43976f7660f-kube-api-access-z5sjd\" (UniqueName: \"kubernetes.io/projected/e799f52e-4fcc-47be-b42c-d43976f7660f-kube-api-access-z5sjd\") pod \"e799f52e-4fcc-47be-b42c-d43976f7660f\" (UID: \"e799f52e-4fcc-47be-b42c-d43976f7660f\") "
Oct 31 13:31:59.098152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.097778    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e799f52e-4fcc-47be-b42c-d43976f7660f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e799f52e-4fcc-47be-b42c-d43976f7660f-maas-api-tls\") pod \"e799f52e-4fcc-47be-b42c-d43976f7660f\" (UID: \"e799f52e-4fcc-47be-b42c-d43976f7660f\") "
Oct 31 13:31:59.098152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.097929    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e799f52e-4fcc-47be-b42c-d43976f7660f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e799f52e-4fcc-47be-b42c-d43976f7660f-openshift-service-ca\") pod \"e799f52e-4fcc-47be-b42c-d43976f7660f\" (UID: \"e799f52e-4fcc-47be-b42c-d43976f7660f\") "
Oct 31 13:31:59.098152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.097987    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e799f52e-4fcc-47be-b42c-d43976f7660f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e799f52e-4fcc-47be-b42c-d43976f7660f-metrics-certs\") pod \"e799f52e-4fcc-47be-b42c-d43976f7660f\" (UID: \"e799f52e-4fcc-47be-b42c-d43976f7660f\") "
Oct 31 13:31:59.099291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.099259    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e799f52e-4fcc-47be-b42c-d43976f7660f-openshift-service-ca" pod "e799f52e-4fcc-47be-b42c-d43976f7660f" (UID: "e799f52e-4fcc-47be-b42c-d43976f7660f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:31:59.123569 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.122167    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e799f52e-4fcc-47be-b42c-d43976f7660f-metrics-certs" pod "e799f52e-4fcc-47be-b42c-d43976f7660f" (UID: "e799f52e-4fcc-47be-b42c-d43976f7660f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:31:59.123569 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.122260    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e799f52e-4fcc-47be-b42c-d43976f7660f-maas-api-tls" pod "e799f52e-4fcc-47be-b42c-d43976f7660f" (UID: "e799f52e-4fcc-47be-b42c-d43976f7660f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:31:59.123569 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.122360    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e799f52e-4fcc-47be-b42c-d43976f7660f-kube-api-access-z5sjd" pod "e799f52e-4fcc-47be-b42c-d43976f7660f" (UID: "e799f52e-4fcc-47be-b42c-d43976f7660f"). InnerVolumeSpecName "kube-api-access-z5sjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:31:59.200909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.200418    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e799f52e-4fcc-47be-b42c-d43976f7660f-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:59.200909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.200462    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e799f52e-4fcc-47be-b42c-d43976f7660f-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:59.200909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.200478    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e799f52e-4fcc-47be-b42c-d43976f7660f-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:59.200909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.200492    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5sjd\" (UniqueName: \"kubernetes.io/projected/e799f52e-4fcc-47be-b42c-d43976f7660f-kube-api-access-z5sjd\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:59.806832 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.806778    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:31:59.870775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.870732    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb" event={"ID":"e799f52e-4fcc-47be-b42c-d43976f7660f","Type":"ContainerDied","Data":"161b06f5ef705e7ea82443fdc4528f59ca86d420593d44d69b7bc70f506fd3e6"}
Oct 31 13:31:59.871126 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.871099    4428 scope.go:122] "RemoveContainer" containerID="4b81cf0cd81b0f911ab21567935b50ca649be412b4e9e14fcf506e3ee56227fa"
Oct 31 13:31:59.871474 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:31:59.871452    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"
Oct 31 13:32:00.165036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:00.164998    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"]
Oct 31 13:32:00.282395 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:00.282342    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a66ae960-69b7c5b4f5-7gcrb"]
Oct 31 13:32:00.733500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:00.733461    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:00.821896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:00.808570    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:32:00.821896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:00.811005    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e799f52e-4fcc-47be-b42c-d43976f7660f" path="/var/lib/kubelet/pods/e799f52e-4fcc-47be-b42c-d43976f7660f/volumes"
Oct 31 13:32:00.939371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:00.939335    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:00.956007 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:00.955969    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:32:01.001055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:01.000958    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:01.117104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:01.117056    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:01.148085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:01.147997    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"]
Oct 31 13:32:01.199021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:01.198983    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f692a8f5-d8dqm"]
Oct 31 13:32:01.214030 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:01.213988    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:01.355951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:01.337959    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:01.357255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:01.356234    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:01.744898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:01.740790    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:01.810046 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:01.809998    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:01.811115 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:01.811090    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:32:02.080118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:01.986714    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:02.102909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:02.052725    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:02.102909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:02.098839    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:32:02.180440 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:02.175059    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-n56bm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:32:02.180440 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:02.175142    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm" podUID="aaa34a48-48ac-4d05-8d9d-0d0c0a870f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:02.310893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:02.279045    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:02.310893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:02.279561    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:32:02.368483 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:02.368386    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:02.622221 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:02.622052    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:32:02.724929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:02.633667    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:02.788923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:02.746343    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:32:03.103900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.081100    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:32:03.282643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.084631    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:32:03.285257 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.124930    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:32:03.285834 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.125424    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:32:03.285991 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.168290    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:03.286116 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.286080    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:32:03.361527 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.361441    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a73586db-b1bb-4629-9357-aacc597c436f" path="/var/lib/kubelet/pods/a73586db-b1bb-4629-9357-aacc597c436f/volumes"
Oct 31 13:32:03.362753 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.362732    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:32:03.416797 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.416766    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:32:03.417400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.417379    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:32:03.417582 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.417570    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:32:03.436765 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.436744    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:32:03.444970 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.444859    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:03.501293 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.500978    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"]
Oct 31 13:32:03.507699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.502108    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b5b7ad21-f603-492f-8096-89922a09b4ce" containerName="revoke-keys"
Oct 31 13:32:03.507699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.502129    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b7ad21-f603-492f-8096-89922a09b4ce" containerName="revoke-keys"
Oct 31 13:32:03.507699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.502149    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e799f52e-4fcc-47be-b42c-d43976f7660f" containerName="maas-api"
Oct 31 13:32:03.507699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.502156    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e799f52e-4fcc-47be-b42c-d43976f7660f" containerName="maas-api"
Oct 31 13:32:03.507699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.502184    4428 cpu_manager.go:414] "containerMap: removing container" podUID="1004aeb3-63d7-48e5-ac93-59eea0a4c66a" containerName="payload-processing"
Oct 31 13:32:03.507699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.502191    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1004aeb3-63d7-48e5-ac93-59eea0a4c66a" containerName="payload-processing"
Oct 31 13:32:03.507699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.502364    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1004aeb3-63d7-48e5-ac93-59eea0a4c66a" containerName="payload-processing"
Oct 31 13:32:03.507699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.502392    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b7ad21-f603-492f-8096-89922a09b4ce" containerName="revoke-keys"
Oct 31 13:32:03.507699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.502404    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e799f52e-4fcc-47be-b42c-d43976f7660f" containerName="maas-api"
Oct 31 13:32:03.512431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.512138    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"
Oct 31 13:32:03.518097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.518060    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:32:03.519813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.519786    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:32:03.520034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.520008    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:32:03.520847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.520826    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:32:03.523367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.523347    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"]
Oct 31 13:32:03.537511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.537471    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"]
Oct 31 13:32:03.542331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.542311    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78cc6525-gtvcn"]
Oct 31 13:32:03.678500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.678414    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90d7ee51-j2nfl\" (UID: \"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"
Oct 31 13:32:03.678728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.678704    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkmss\" (UniqueName: \"kubernetes.io/projected/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-kube-api-access-xkmss\") pod \"maas-api-revoke-sub-90d7ee51-j2nfl\" (UID: \"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"
Oct 31 13:32:03.678995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.678972    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90d7ee51-j2nfl\" (UID: \"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"
Oct 31 13:32:03.682016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.681986    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:03.781438 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.781402    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90d7ee51-j2nfl\" (UID: \"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"
Oct 31 13:32:03.781613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.781448    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkmss\" (UniqueName: \"kubernetes.io/projected/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-kube-api-access-xkmss\") pod \"maas-api-revoke-sub-90d7ee51-j2nfl\" (UID: \"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"
Oct 31 13:32:03.781613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.781552    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90d7ee51-j2nfl\" (UID: \"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"
Oct 31 13:32:03.782464 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.782428    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90d7ee51-j2nfl\" (UID: \"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"
Oct 31 13:32:03.785781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.785757    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90d7ee51-j2nfl\" (UID: \"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"
Oct 31 13:32:03.795274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.795245    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkmss\" (UniqueName: \"kubernetes.io/projected/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-kube-api-access-xkmss\") pod \"maas-api-revoke-sub-90d7ee51-j2nfl\" (UID: \"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"
Oct 31 13:32:03.835966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:03.835935    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"
Oct 31 13:32:04.124014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:04.123985    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"]
Oct 31 13:32:04.129386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:04.129364    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"
Oct 31 13:32:04.140919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:04.140148    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"]
Oct 31 13:32:04.185273 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:32:04.185241    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aeeeb1c_3cec_4d8a_b7c4_edfaf70bfefd.slice/crio-2a7378beecb88565a0af626f899c034404c2cb0ad7c340c905a00d35731a9433 WatchSource:0}: Error finding container 2a7378beecb88565a0af626f899c034404c2cb0ad7c340c905a00d35731a9433: Status 404 returned error can't find the container with id 2a7378beecb88565a0af626f899c034404c2cb0ad7c340c905a00d35731a9433
Oct 31 13:32:04.196975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:04.196912    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"]
Oct 31 13:32:04.265703 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:04.240421    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:04.305894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:04.297116    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-728deca0-shldl\" (UID: \"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"
Oct 31 13:32:04.305894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:04.297414    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbnr9\" (UniqueName: \"kubernetes.io/projected/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-kube-api-access-tbnr9\") pod \"maas-api-revoke-sub-728deca0-shldl\" (UID: \"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"
Oct 31 13:32:04.305894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:04.297511    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-728deca0-shldl\" (UID: \"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"
Oct 31 13:32:04.422902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:04.421533    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-728deca0-shldl\" (UID: \"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"
Oct 31 13:32:04.422902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:04.421705    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-728deca0-shldl\" (UID: \"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"
Oct 31 13:32:04.422902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:04.422014    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbnr9\" (UniqueName: \"kubernetes.io/projected/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-kube-api-access-tbnr9\") pod \"maas-api-revoke-sub-728deca0-shldl\" (UID: \"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"
Oct 31 13:32:04.423669 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:04.423538    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-728deca0-shldl\" (UID: \"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"
Oct 31 13:32:04.441965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:04.441931    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl" event={"ID":"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd","Type":"ContainerStarted","Data":"2a7378beecb88565a0af626f899c034404c2cb0ad7c340c905a00d35731a9433"}
Oct 31 13:32:04.448777 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:04.448670    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbnr9\" (UniqueName: \"kubernetes.io/projected/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-kube-api-access-tbnr9\") pod \"maas-api-revoke-sub-728deca0-shldl\" (UID: \"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"
Oct 31 13:32:04.519217 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:04.518595    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-728deca0-shldl\" (UID: \"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"
Oct 31 13:32:04.779769 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:04.779741    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"
Oct 31 13:32:04.780551 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:04.780522    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c69f8f7-f106-4c54-ad97-616218cb4ab8" path="/var/lib/kubelet/pods/0c69f8f7-f106-4c54-ad97-616218cb4ab8/volumes"
Oct 31 13:32:05.093432 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:32:05.093401    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1a7f6d_f8a7_4a29_81e2_f4bb51498fe4.slice/crio-5d296edd046ba02a7a6be1dac8f9a6d130dbedd7e14fdcdee3c4409b3525718f WatchSource:0}: Error finding container 5d296edd046ba02a7a6be1dac8f9a6d130dbedd7e14fdcdee3c4409b3525718f: Status 404 returned error can't find the container with id 5d296edd046ba02a7a6be1dac8f9a6d130dbedd7e14fdcdee3c4409b3525718f
Oct 31 13:32:05.093583 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:05.093559    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"]
Oct 31 13:32:05.224906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:05.224774    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:05.482092 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:05.482058    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl" event={"ID":"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd","Type":"ContainerStarted","Data":"5942299c823b74cff32f19482b9cfd811eddf7d92a15554c8bba531cd724b9bc"}
Oct 31 13:32:05.483770 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:05.483749    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl" event={"ID":"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4","Type":"ContainerStarted","Data":"5d296edd046ba02a7a6be1dac8f9a6d130dbedd7e14fdcdee3c4409b3525718f"}
Oct 31 13:32:05.516566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:05.516514    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl" podStartSLOduration=3.516496153 podStartE2EDuration="3.516496153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:32:05.510060542 +0000 UTC m=+1722.908409167" watchObservedRunningTime="2026-10-31 13:32:05.516496153 +0000 UTC m=+1722.914844778"
Oct 31 13:32:06.261988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:06.256016    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:06.507001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:06.506965    4428 generic.go:356] "Generic (PLEG): container finished" podID="1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4" containerID="b4bf13ff74b9c112e8a5f9684c984f4fb9089a7a0ef94130e3458e70f7368024" exitCode=0
Oct 31 13:32:06.507643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:06.507617    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl" event={"ID":"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4","Type":"ContainerDied","Data":"b4bf13ff74b9c112e8a5f9684c984f4fb9089a7a0ef94130e3458e70f7368024"}
Oct 31 13:32:06.509775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:06.509745    4428 generic.go:356] "Generic (PLEG): container finished" podID="0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd" containerID="5942299c823b74cff32f19482b9cfd811eddf7d92a15554c8bba531cd724b9bc" exitCode=0
Oct 31 13:32:06.509931 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:06.509839    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl" event={"ID":"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd","Type":"ContainerDied","Data":"5942299c823b74cff32f19482b9cfd811eddf7d92a15554c8bba531cd724b9bc"}
Oct 31 13:32:06.914002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:06.913916    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:07.285894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:07.281922    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"]
Oct 31 13:32:07.377925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:07.374050    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dff70365-mm4vz"]
Oct 31 13:32:07.475954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:07.474415    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:07.736596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:07.736308    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:07.855537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:07.855300    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"]
Oct 31 13:32:07.856942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:07.856214    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw" podUID="af8ebab8-5e74-42e6-ac89-b00164117531" containerName="istio-proxy" containerID="cri-o://7f18665c48ce6dcb1a5cfc6aed48591ed9c96d5590fe3b22837cc579ca257910" gracePeriod=30
Oct 31 13:32:07.959521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:07.947820    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:08.394625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.393000    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"
Oct 31 13:32:08.399653 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.395925    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"
Oct 31 13:32:08.477391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.471407    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-kube-api-access-xkmss\" (UniqueName: \"kubernetes.io/projected/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-kube-api-access-xkmss\") pod \"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd\" (UID: \"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd\") "
Oct 31 13:32:08.477391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.471517    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-maas-api-xks-ca\") pod \"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd\" (UID: \"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd\") "
Oct 31 13:32:08.477391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.471611    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-maas-api-openshift-ca\") pod \"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4\" (UID: \"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4\") "
Oct 31 13:32:08.477391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.471755    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-maas-api-xks-ca\") pod \"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4\" (UID: \"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4\") "
Oct 31 13:32:08.477391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.471828    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-kube-api-access-tbnr9\" (UniqueName: \"kubernetes.io/projected/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-kube-api-access-tbnr9\") pod \"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4\" (UID: \"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4\") "
Oct 31 13:32:08.477391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.471894    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-maas-api-openshift-ca\") pod \"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd\" (UID: \"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd\") "
Oct 31 13:32:08.477391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.473758    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-maas-api-openshift-ca" pod "1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4" (UID: "1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:32:08.477391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.474153    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-maas-api-openshift-ca" pod "0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd" (UID: "0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:32:08.477391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.474983    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-maas-api-xks-ca" pod "0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd" (UID: "0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:32:08.477391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.477186    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-kube-api-access-xkmss" pod "0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd" (UID: "0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd"). InnerVolumeSpecName "kube-api-access-xkmss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:32:08.481907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.478463    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-kube-api-access-tbnr9" pod "1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4" (UID: "1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4"). InnerVolumeSpecName "kube-api-access-tbnr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:32:08.481907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.480574    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-maas-api-xks-ca" pod "1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4" (UID: "1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:32:08.573858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.573781    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbnr9\" (UniqueName: \"kubernetes.io/projected/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-kube-api-access-tbnr9\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:08.573858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.573812    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:08.573858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.573822    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkmss\" (UniqueName: \"kubernetes.io/projected/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-kube-api-access-xkmss\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:08.573858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.573836    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:08.573858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.573851    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:08.573858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.573876    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:08.642905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.633901    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:08.751217 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.751180    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"
Oct 31 13:32:08.758920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.755018    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"
Oct 31 13:32:08.769268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.769237    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5b7ad21-f603-492f-8096-89922a09b4ce" path="/var/lib/kubelet/pods/b5b7ad21-f603-492f-8096-89922a09b4ce/volumes"
Oct 31 13:32:08.770273 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.770246    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl" event={"ID":"0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd","Type":"ContainerDied","Data":"2a7378beecb88565a0af626f899c034404c2cb0ad7c340c905a00d35731a9433"}
Oct 31 13:32:08.770406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.770291    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a7378beecb88565a0af626f899c034404c2cb0ad7c340c905a00d35731a9433"
Oct 31 13:32:08.770406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.770301    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl" event={"ID":"1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4","Type":"ContainerDied","Data":"5d296edd046ba02a7a6be1dac8f9a6d130dbedd7e14fdcdee3c4409b3525718f"}
Oct 31 13:32:08.770406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.770310    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d296edd046ba02a7a6be1dac8f9a6d130dbedd7e14fdcdee3c4409b3525718f"
Oct 31 13:32:08.876782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:08.876685    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:09.067605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:09.067568    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:09.237039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:09.236998    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:09.663939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:09.663710    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:10.894969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:10.888348    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:11.055075 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:11.055045    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"]
Oct 31 13:32:11.059779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:11.059758    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69f67ed7-z55vx"]
Oct 31 13:32:11.811892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:11.810636    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:12.769206 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.769117    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:12.790651 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.790613    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdb1ec4c-74bd-4137-a4c5-a3ce25638f21" path="/var/lib/kubelet/pods/bdb1ec4c-74bd-4137-a4c5-a3ce25638f21/volumes"
Oct 31 13:32:12.801352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.801322    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:32:12.872779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.872750    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-istio-data\") pod \"af8ebab8-5e74-42e6-ac89-b00164117531\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") "
Oct 31 13:32:12.872986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.872807    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af8ebab8-5e74-42e6-ac89-b00164117531-istio-token\" (UniqueName: \"kubernetes.io/projected/af8ebab8-5e74-42e6-ac89-b00164117531-istio-token\") pod \"af8ebab8-5e74-42e6-ac89-b00164117531\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") "
Oct 31 13:32:12.873120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.873074    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-istio-data" pod "af8ebab8-5e74-42e6-ac89-b00164117531" (UID: "af8ebab8-5e74-42e6-ac89-b00164117531"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:32:12.873266 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.873135    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-istio-envoy\") pod \"af8ebab8-5e74-42e6-ac89-b00164117531\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") "
Oct 31 13:32:12.873266 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.873229    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af8ebab8-5e74-42e6-ac89-b00164117531-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af8ebab8-5e74-42e6-ac89-b00164117531-istio-podinfo\") pod \"af8ebab8-5e74-42e6-ac89-b00164117531\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") "
Oct 31 13:32:12.873385 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.873304    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-workload-socket\") pod \"af8ebab8-5e74-42e6-ac89-b00164117531\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") "
Oct 31 13:32:12.873444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.873416    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-credential-socket\") pod \"af8ebab8-5e74-42e6-ac89-b00164117531\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") "
Oct 31 13:32:12.873511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.873500    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af8ebab8-5e74-42e6-ac89-b00164117531-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af8ebab8-5e74-42e6-ac89-b00164117531-istiod-ca-cert\") pod \"af8ebab8-5e74-42e6-ac89-b00164117531\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") "
Oct 31 13:32:12.873568 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.873525    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af8ebab8-5e74-42e6-ac89-b00164117531-kube-api-access-sktq9\" (UniqueName: \"kubernetes.io/projected/af8ebab8-5e74-42e6-ac89-b00164117531-kube-api-access-sktq9\") pod \"af8ebab8-5e74-42e6-ac89-b00164117531\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") "
Oct 31 13:32:12.873731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.873713    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-workload-certs\") pod \"af8ebab8-5e74-42e6-ac89-b00164117531\" (UID: \"af8ebab8-5e74-42e6-ac89-b00164117531\") "
Oct 31 13:32:12.873846 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.873816    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-workload-socket" pod "af8ebab8-5e74-42e6-ac89-b00164117531" (UID: "af8ebab8-5e74-42e6-ac89-b00164117531"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:32:12.874660 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.874632    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:12.875002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.874663    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:12.875002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.874934    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-workload-certs" pod "af8ebab8-5e74-42e6-ac89-b00164117531" (UID: "af8ebab8-5e74-42e6-ac89-b00164117531"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:32:12.875139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.875125    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-credential-socket" pod "af8ebab8-5e74-42e6-ac89-b00164117531" (UID: "af8ebab8-5e74-42e6-ac89-b00164117531"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:32:12.875494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.875440    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af8ebab8-5e74-42e6-ac89-b00164117531-istiod-ca-cert" pod "af8ebab8-5e74-42e6-ac89-b00164117531" (UID: "af8ebab8-5e74-42e6-ac89-b00164117531"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:32:12.889661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.886507    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af8ebab8-5e74-42e6-ac89-b00164117531-istio-token" pod "af8ebab8-5e74-42e6-ac89-b00164117531" (UID: "af8ebab8-5e74-42e6-ac89-b00164117531"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:32:12.889661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.887813    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af8ebab8-5e74-42e6-ac89-b00164117531-istio-podinfo" pod "af8ebab8-5e74-42e6-ac89-b00164117531" (UID: "af8ebab8-5e74-42e6-ac89-b00164117531"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:32:12.890759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.890705    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-istio-envoy" pod "af8ebab8-5e74-42e6-ac89-b00164117531" (UID: "af8ebab8-5e74-42e6-ac89-b00164117531"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:32:12.891257 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.891232    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af8ebab8-5e74-42e6-ac89-b00164117531-kube-api-access-sktq9" pod "af8ebab8-5e74-42e6-ac89-b00164117531" (UID: "af8ebab8-5e74-42e6-ac89-b00164117531"). InnerVolumeSpecName "kube-api-access-sktq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:32:12.903788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.903759    4428 generic.go:356] "Generic (PLEG): container finished" podID="af8ebab8-5e74-42e6-ac89-b00164117531" containerID="7f18665c48ce6dcb1a5cfc6aed48591ed9c96d5590fe3b22837cc579ca257910" exitCode=0
Oct 31 13:32:12.903980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.903812    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"
Oct 31 13:32:12.903980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.903851    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw" event={"ID":"af8ebab8-5e74-42e6-ac89-b00164117531","Type":"ContainerDied","Data":"7f18665c48ce6dcb1a5cfc6aed48591ed9c96d5590fe3b22837cc579ca257910"}
Oct 31 13:32:12.903980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.903940    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw" event={"ID":"af8ebab8-5e74-42e6-ac89-b00164117531","Type":"ContainerDied","Data":"19354a81543c35af1c77f9de420670fb112f64d9aeaf090dbb68656f6ec2f6fb"}
Oct 31 13:32:12.903980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.903975    4428 scope.go:122] "RemoveContainer" containerID="7f18665c48ce6dcb1a5cfc6aed48591ed9c96d5590fe3b22837cc579ca257910"
Oct 31 13:32:12.917664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.917644    4428 scope.go:122] "RemoveContainer" containerID="7f18665c48ce6dcb1a5cfc6aed48591ed9c96d5590fe3b22837cc579ca257910"
Oct 31 13:32:12.918074 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:32:12.918017    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f18665c48ce6dcb1a5cfc6aed48591ed9c96d5590fe3b22837cc579ca257910\": container with ID starting with 7f18665c48ce6dcb1a5cfc6aed48591ed9c96d5590fe3b22837cc579ca257910 not found: ID does not exist" containerID="7f18665c48ce6dcb1a5cfc6aed48591ed9c96d5590fe3b22837cc579ca257910"
Oct 31 13:32:12.918214 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.918078    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f18665c48ce6dcb1a5cfc6aed48591ed9c96d5590fe3b22837cc579ca257910"} err="failed to get container status \"7f18665c48ce6dcb1a5cfc6aed48591ed9c96d5590fe3b22837cc579ca257910\": rpc error: code = NotFound desc = could not find container \"7f18665c48ce6dcb1a5cfc6aed48591ed9c96d5590fe3b22837cc579ca257910\": container with ID starting with 7f18665c48ce6dcb1a5cfc6aed48591ed9c96d5590fe3b22837cc579ca257910 not found: ID does not exist"
Oct 31 13:32:12.934456 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.934425    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"]
Oct 31 13:32:12.940331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.940303    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:12.941052 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.941034    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-dff70365-openshift-default-7b549865c5-f7zxw"]
Oct 31 13:32:12.976379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.976355    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af8ebab8-5e74-42e6-ac89-b00164117531-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:12.976467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.976383    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:12.976467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.976399    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af8ebab8-5e74-42e6-ac89-b00164117531-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:12.976467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.976408    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:12.976467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.976417    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af8ebab8-5e74-42e6-ac89-b00164117531-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:12.976467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.976425    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sktq9\" (UniqueName: \"kubernetes.io/projected/af8ebab8-5e74-42e6-ac89-b00164117531-kube-api-access-sktq9\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:12.976467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:12.976435    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af8ebab8-5e74-42e6-ac89-b00164117531-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:13.099646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:13.099618    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:13.715889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:13.708378    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:14.450103 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:14.450076    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:14.791894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:14.791793    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:14.818274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:14.817704    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af8ebab8-5e74-42e6-ac89-b00164117531" path="/var/lib/kubelet/pods/af8ebab8-5e74-42e6-ac89-b00164117531/volumes"
Oct 31 13:32:14.971922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:14.968489    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:15.314693 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:15.314658    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:15.489236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:15.489190    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:15.862152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:15.862111    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:16.060985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:16.060946    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"]
Oct 31 13:32:16.061282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:16.061261    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-807497f4-nh5x5"]
Oct 31 13:32:16.082456 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:16.082420    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:16.758535 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:16.758501    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f336286-4447-42e0-98be-cb39b7f0a4ee" path="/var/lib/kubelet/pods/3f336286-4447-42e0-98be-cb39b7f0a4ee/volumes"
Oct 31 13:32:17.024543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:17.024444    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:17.450223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:17.450155    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:17.617528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:17.617493    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:18.017897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:18.017439    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:18.740406 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:32:18.740371    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:32:19.034743 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:19.034651    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:19.301574 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:19.301488    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:19.497753 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:19.497721    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:20.132590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:20.132116    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:20.684962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:20.684880    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:20.964952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:20.960741    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:21.167346 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.167315    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:21.307353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.307280    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"]
Oct 31 13:32:21.308504 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.308480    4428 cpu_manager.go:414] "containerMap: removing container" podUID="af8ebab8-5e74-42e6-ac89-b00164117531" containerName="istio-proxy"
Oct 31 13:32:21.308602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.308509    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af8ebab8-5e74-42e6-ac89-b00164117531" containerName="istio-proxy"
Oct 31 13:32:21.308602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.308526    4428 cpu_manager.go:414] "containerMap: removing container" podUID="0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd" containerName="revoke-keys"
Oct 31 13:32:21.308602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.308537    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd" containerName="revoke-keys"
Oct 31 13:32:21.308602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.308565    4428 cpu_manager.go:414] "containerMap: removing container" podUID="1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4" containerName="revoke-keys"
Oct 31 13:32:21.308602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.308574    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4" containerName="revoke-keys"
Oct 31 13:32:21.308817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.308752    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="af8ebab8-5e74-42e6-ac89-b00164117531" containerName="istio-proxy"
Oct 31 13:32:21.308817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.308768    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd" containerName="revoke-keys"
Oct 31 13:32:21.308817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.308784    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4" containerName="revoke-keys"
Oct 31 13:32:21.313657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.313633    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.321565 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.321541    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-79f4fb0f-openshift-default-dockercfg-shgv6\""
Oct 31 13:32:21.325372 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.325330    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"]
Oct 31 13:32:21.361427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.361391    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:21.411245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.411213    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.411407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.411260    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.411407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.411278    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.411526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.411454    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.411599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.411544    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbcsq\" (UniqueName: \"kubernetes.io/projected/05c3c622-9d4a-4391-bb25-e07daa031d3b-kube-api-access-jbcsq\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.411713 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.411670    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.411773 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.411738    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.411811 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.411798    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c3c622-9d4a-4391-bb25-e07daa031d3b-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.411858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.411845    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.514177 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.514140    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.515167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.515139    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.515735 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.515717    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.516210 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.516193    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.516771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.516753    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.521493 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.521473    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbcsq\" (UniqueName: \"kubernetes.io/projected/05c3c622-9d4a-4391-bb25-e07daa031d3b-kube-api-access-jbcsq\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.521722 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.521705    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.530417 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.530394    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.530937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.530918    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c3c622-9d4a-4391-bb25-e07daa031d3b-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.536764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.522137    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.536959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.516141    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.537073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.530814    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.544585 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.515644    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.544733 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.531963    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c3c622-9d4a-4391-bb25-e07daa031d3b-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.564755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.564676    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:21.575625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.566773    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.575904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.567158    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.617038 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.613472    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.633887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.617499    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbcsq\" (UniqueName: \"kubernetes.io/projected/05c3c622-9d4a-4391-bb25-e07daa031d3b-kube-api-access-jbcsq\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:21.634095 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:21.634006    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:22.173693 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:22.173619    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"]
Oct 31 13:32:22.174655 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:32:22.174630    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c3c622_9d4a_4391_bb25_e07daa031d3b.slice/crio-62a79ed32f6517f81caf53dfc74b4e2ccbb0246323aea0da79e3badb44799089 WatchSource:0}: Error finding container 62a79ed32f6517f81caf53dfc74b4e2ccbb0246323aea0da79e3badb44799089: Status 404 returned error can't find the container with id 62a79ed32f6517f81caf53dfc74b4e2ccbb0246323aea0da79e3badb44799089
Oct 31 13:32:22.178011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:22.177973    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:32:22.178093 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:22.178060    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:32:22.178133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:22.178106    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:32:22.456043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:22.455692    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:23.115784 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:23.115744    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz" event={"ID":"05c3c622-9d4a-4391-bb25-e07daa031d3b","Type":"ContainerStarted","Data":"d878d22f219f9da551c49546ba7589ec1ed1c0d1884cf56056ec8fa180e6f05a"}
Oct 31 13:32:23.115992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:23.115805    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz" event={"ID":"05c3c622-9d4a-4391-bb25-e07daa031d3b","Type":"ContainerStarted","Data":"62a79ed32f6517f81caf53dfc74b4e2ccbb0246323aea0da79e3badb44799089"}
Oct 31 13:32:23.152737 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:23.152681    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz" podStartSLOduration=2.152667324 podStartE2EDuration="2.152667324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:32:23.149367769 +0000 UTC m=+1740.547716393" watchObservedRunningTime="2026-10-31 13:32:23.152667324 +0000 UTC m=+1740.551015948"
Oct 31 13:32:23.307894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:23.306926    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:23.481777 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:23.481741    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:23.635075 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:23.635043    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:23.640451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:23.640428    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:24.124360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:24.124331    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:24.125313 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:24.125291    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:32:24.241836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:24.241808    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:25.251854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:25.249787    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:25.440814 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:25.440733    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:25.603463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:25.603428    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:26.145896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:26.135681    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:26.730207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:26.730147    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:26.963892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:26.949491    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:27.304018 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:27.303966    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:28.333732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:28.333700    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:28.507752 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:28.507718    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:28.674169 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:28.674084    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:29.299136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:29.299105    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:29.540891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:29.532387    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:30.389894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:30.384407    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:31.148927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:31.148854    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:32.009896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:32.007731    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:32.926710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:32.926676    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:33.827823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:33.825053    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:34.849100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:34.848840    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:36.121666 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:36.121631    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:36.508964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:36.500194    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:37.542747 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:37.542720    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:38.012462 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:38.012430    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:39.189114 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:39.189053    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:40.019033 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:40.018981    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:40.924903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:40.924847    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:41.888454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:41.888394    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:42.825674 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:42.825628    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:43.032713 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:43.032682    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"]
Oct 31 13:32:43.037433 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:43.037408    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3207b368-x89c8"]
Oct 31 13:32:43.605191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:43.605103    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:44.338519 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:44.338484    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:44.518489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:44.518453    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:44.757522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:44.757487    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414a88f7-a4a5-45fa-a922-46987a48d81c" path="/var/lib/kubelet/pods/414a88f7-a4a5-45fa-a922-46987a48d81c/volumes"
Oct 31 13:32:45.347621 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:45.347581    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:45.719825 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:45.719793    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:45.905209 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:45.905181    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:46.448240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:46.447985    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:47.909321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:47.909287    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:48.783481 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:48.783431    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:48.803303 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:32:48.803266    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:32:49.795118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:49.795064    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:50.790455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:50.790389    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:50.986072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:50.986042    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:51.597665 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:51.597568    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:52.029297 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:52.029254    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"]
Oct 31 13:32:52.038619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:52.038586    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c514b01f-slkxh"]
Oct 31 13:32:52.615963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:52.615842    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:52.767966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:52.767931    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c98b0b-a08c-453a-a83f-2605f568ef8d" path="/var/lib/kubelet/pods/79c98b0b-a08c-453a-a83f-2605f568ef8d/volumes"
Oct 31 13:32:53.102430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:53.102395    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:53.270941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:53.270862    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:53.849980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:53.849851    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:54.819915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:54.819828    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:55.885977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:55.885898    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:56.091915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:56.091824    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:56.550277 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:56.550242    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:56.759618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:56.759581    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:56.959375 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:56.959291    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:57.853007 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:57.852971    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:58.861890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:58.858499    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:32:59.882226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:32:59.882188    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:00.955694 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:00.955651    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:01.894850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:01.894801    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:02.924339 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:02.924300    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:04.084412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:04.084373    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:05.162601 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:05.162540    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:06.235548 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:06.235509    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:07.229846 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:07.229802    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:08.124811 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:08.124774    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:09.120456 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:09.120416    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:10.021923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:10.021781    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:10.831945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:10.831887    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:11.732279 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:11.730080    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:12.483064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:12.482287    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:13.060252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:13.060204    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:13.496998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:13.496945    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:14.306802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:14.306767    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:15.110842 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:15.110761    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:15.292264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:15.292229    4428 generic.go:356] "Generic (PLEG): container finished" podID="05c3c622-9d4a-4391-bb25-e07daa031d3b" containerID="d878d22f219f9da551c49546ba7589ec1ed1c0d1884cf56056ec8fa180e6f05a" exitCode=0
Oct 31 13:33:15.292537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:15.292512    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz" event={"ID":"05c3c622-9d4a-4391-bb25-e07daa031d3b","Type":"ContainerDied","Data":"d878d22f219f9da551c49546ba7589ec1ed1c0d1884cf56056ec8fa180e6f05a"}
Oct 31 13:33:15.293128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:15.293103    4428 scope.go:122] "RemoveContainer" containerID="d878d22f219f9da551c49546ba7589ec1ed1c0d1884cf56056ec8fa180e6f05a"
Oct 31 13:33:15.295079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:15.295048    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:33:15.295178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:15.295122    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:33:15.295178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:15.295153    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:33:15.999900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:15.997588    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:16.064893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:16.064796    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"]
Oct 31 13:33:16.089901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:16.086747    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d68719-l2k98"]
Oct 31 13:33:16.304053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:16.303959    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz" event={"ID":"05c3c622-9d4a-4391-bb25-e07daa031d3b","Type":"ContainerStarted","Data":"c6436d1bd8b649a7275752cb416dbd0d827de5c5426a882c80d1df4da4ae5703"}
Oct 31 13:33:16.635178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:16.635092    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:33:16.635356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:16.635240    4428 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 13:33:16.635356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:16.635289    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz" podUID="05c3c622-9d4a-4391-bb25-e07daa031d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 13:33:16.780584 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:16.780551    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d97567d5-0707-423e-bb4c-213ceb96f80e" path="/var/lib/kubelet/pods/d97567d5-0707-423e-bb4c-213ceb96f80e/volumes"
Oct 31 13:33:16.852671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:16.852633    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:17.128060 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.128023    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"]
Oct 31 13:33:17.132877 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.132843    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"
Oct 31 13:33:17.141068 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.141047    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:33:17.141439 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.141420    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:33:17.184437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.184407    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"]
Oct 31 13:33:17.228509 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.228476    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7\" (UID: \"ea5176c0-4a26-4347-8bcb-8411e8c98f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"
Oct 31 13:33:17.228687 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.228522    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2ql2\" (UniqueName: \"kubernetes.io/projected/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-kube-api-access-v2ql2\") pod \"maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7\" (UID: \"ea5176c0-4a26-4347-8bcb-8411e8c98f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"
Oct 31 13:33:17.228687 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.228663    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7\" (UID: \"ea5176c0-4a26-4347-8bcb-8411e8c98f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"
Oct 31 13:33:17.329957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.329921    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7\" (UID: \"ea5176c0-4a26-4347-8bcb-8411e8c98f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"
Oct 31 13:33:17.329957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.329961    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ql2\" (UniqueName: \"kubernetes.io/projected/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-kube-api-access-v2ql2\") pod \"maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7\" (UID: \"ea5176c0-4a26-4347-8bcb-8411e8c98f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"
Oct 31 13:33:17.330218 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.330025    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7\" (UID: \"ea5176c0-4a26-4347-8bcb-8411e8c98f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"
Oct 31 13:33:17.330759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.330672    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7\" (UID: \"ea5176c0-4a26-4347-8bcb-8411e8c98f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"
Oct 31 13:33:17.332478 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.332456    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7\" (UID: \"ea5176c0-4a26-4347-8bcb-8411e8c98f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"
Oct 31 13:33:17.372321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.372297    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2ql2\" (UniqueName: \"kubernetes.io/projected/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-kube-api-access-v2ql2\") pod \"maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7\" (UID: \"ea5176c0-4a26-4347-8bcb-8411e8c98f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"
Oct 31 13:33:17.473422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.473393    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"
Oct 31 13:33:17.635943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.635909    4428 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 13:33:17.636125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.635966    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz" podUID="05c3c622-9d4a-4391-bb25-e07daa031d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 13:33:17.775312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.775279    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"]
Oct 31 13:33:17.776491 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:33:17.776464    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5176c0_4a26_4347_8bcb_8411e8c98f4f.slice/crio-ebd5e09eaf0d53b5db5013f9fc975e401afe7c0f0b7d2ece52a4146f9662a9d8 WatchSource:0}: Error finding container ebd5e09eaf0d53b5db5013f9fc975e401afe7c0f0b7d2ece52a4146f9662a9d8: Status 404 returned error can't find the container with id ebd5e09eaf0d53b5db5013f9fc975e401afe7c0f0b7d2ece52a4146f9662a9d8
Oct 31 13:33:17.839890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:17.832284    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:18.173577 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:18.173530    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:18.323275 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:18.323236    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7" event={"ID":"ea5176c0-4a26-4347-8bcb-8411e8c98f4f","Type":"ContainerStarted","Data":"20cf6d9bcdd41646e1967fe5e6a849a7feb37d5400f84c66b62d9939333be5ef"}
Oct 31 13:33:18.323447 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:18.323295    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7" event={"ID":"ea5176c0-4a26-4347-8bcb-8411e8c98f4f","Type":"ContainerStarted","Data":"ebd5e09eaf0d53b5db5013f9fc975e401afe7c0f0b7d2ece52a4146f9662a9d8"}
Oct 31 13:33:18.374349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:18.374289    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7" podStartSLOduration=1.374274345 podStartE2EDuration="1.374274345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:33:18.370154704 +0000 UTC m=+1795.768503328" watchObservedRunningTime="2026-10-31 13:33:18.374274345 +0000 UTC m=+1795.772622969"
Oct 31 13:33:18.635271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:18.635242    4428 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 13:33:18.635445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:18.635289    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz" podUID="05c3c622-9d4a-4391-bb25-e07daa031d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 13:33:18.740892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:18.740746    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:18.926818 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:33:18.926780    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:33:19.332564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:19.332528    4428 generic.go:356] "Generic (PLEG): container finished" podID="ea5176c0-4a26-4347-8bcb-8411e8c98f4f" containerID="20cf6d9bcdd41646e1967fe5e6a849a7feb37d5400f84c66b62d9939333be5ef" exitCode=0
Oct 31 13:33:19.333062 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:19.332622    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7" event={"ID":"ea5176c0-4a26-4347-8bcb-8411e8c98f4f","Type":"ContainerDied","Data":"20cf6d9bcdd41646e1967fe5e6a849a7feb37d5400f84c66b62d9939333be5ef"}
Oct 31 13:33:19.621980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:19.621857    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:19.651890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:19.646804    4428 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 13:33:19.651890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:19.647078    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz" podUID="05c3c622-9d4a-4391-bb25-e07daa031d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 13:33:20.725053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:20.725006    4428 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 13:33:20.725552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:20.725068    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz" podUID="05c3c622-9d4a-4391-bb25-e07daa031d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 13:33:20.737454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:20.737360    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:20.792770 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:20.792747    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"
Oct 31 13:33:20.870454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:20.870420    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-kube-api-access-v2ql2\" (UniqueName: \"kubernetes.io/projected/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-kube-api-access-v2ql2\") pod \"ea5176c0-4a26-4347-8bcb-8411e8c98f4f\" (UID: \"ea5176c0-4a26-4347-8bcb-8411e8c98f4f\") "
Oct 31 13:33:20.870648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:20.870491    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-maas-api-openshift-ca\") pod \"ea5176c0-4a26-4347-8bcb-8411e8c98f4f\" (UID: \"ea5176c0-4a26-4347-8bcb-8411e8c98f4f\") "
Oct 31 13:33:20.870711 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:20.870677    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-maas-api-xks-ca\") pod \"ea5176c0-4a26-4347-8bcb-8411e8c98f4f\" (UID: \"ea5176c0-4a26-4347-8bcb-8411e8c98f4f\") "
Oct 31 13:33:20.870889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:20.870833    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-maas-api-openshift-ca" pod "ea5176c0-4a26-4347-8bcb-8411e8c98f4f" (UID: "ea5176c0-4a26-4347-8bcb-8411e8c98f4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:33:20.871444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:20.871422    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:33:20.873401 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:20.873281    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-kube-api-access-v2ql2" pod "ea5176c0-4a26-4347-8bcb-8411e8c98f4f" (UID: "ea5176c0-4a26-4347-8bcb-8411e8c98f4f"). InnerVolumeSpecName "kube-api-access-v2ql2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:33:20.874045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:20.873670    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-maas-api-xks-ca" pod "ea5176c0-4a26-4347-8bcb-8411e8c98f4f" (UID: "ea5176c0-4a26-4347-8bcb-8411e8c98f4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:33:20.973480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:20.973442    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2ql2\" (UniqueName: \"kubernetes.io/projected/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-kube-api-access-v2ql2\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:33:20.973480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:20.973475    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea5176c0-4a26-4347-8bcb-8411e8c98f4f-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:33:21.331922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:21.327960    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:21.403909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:21.403313    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7" event={"ID":"ea5176c0-4a26-4347-8bcb-8411e8c98f4f","Type":"ContainerDied","Data":"ebd5e09eaf0d53b5db5013f9fc975e401afe7c0f0b7d2ece52a4146f9662a9d8"}
Oct 31 13:33:21.403909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:21.403404    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebd5e09eaf0d53b5db5013f9fc975e401afe7c0f0b7d2ece52a4146f9662a9d8"
Oct 31 13:33:21.403909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:21.403533    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"
Oct 31 13:33:21.682978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:21.682897    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:33:21.683586 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:21.683559    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:33:21.685698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:21.685677    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:33:22.122717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:22.119976    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:22.641381 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:22.641347    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:22.799489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:22.799452    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:22.966437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:22.963836    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:23.718896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:23.718150    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:24.455911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:24.451789    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:25.129042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:25.129013    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:25.300203 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:25.295886    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:25.332098 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:25.332049    4428 scope.go:122] "RemoveContainer" containerID="e58f3fcc4a84107059728062ef52982dfcf8e1416083104f9b3c3ef1d7f18517"
Oct 31 13:33:25.392297 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:25.391916    4428 scope.go:122] "RemoveContainer" containerID="e74cfb2ad407bc5147012b7155e9e655fe04864c2e1c84229e3a8644502bbea8"
Oct 31 13:33:25.467228 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:25.467197    4428 scope.go:122] "RemoveContainer" containerID="c02c3693ec5c0433742a101a17836690b64fd4e9aae34a497803ad6572df2d44"
Oct 31 13:33:25.487784 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:25.487758    4428 scope.go:122] "RemoveContainer" containerID="357b4a2bb43dc82170a490919e3acefb5d955b5e1c744bdf2324e5883e3125c5"
Oct 31 13:33:25.502445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:25.502422    4428 scope.go:122] "RemoveContainer" containerID="e71d4a4910aaa853ba618375921cdba12ded1713b76de349a16af9d6d0bc36fb"
Oct 31 13:33:25.512352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:25.512333    4428 scope.go:122] "RemoveContainer" containerID="a7ff1e8370894abb741f297bf549badbd18312253c31234cbe21d00a1742297b"
Oct 31 13:33:26.440895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:26.433245    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:27.160375 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:27.160335    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"]
Oct 31 13:33:27.283124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:27.283083    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:27.480906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:27.480532    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:27.668313 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:27.668274    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:28.577687 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:28.577653    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:29.321928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:29.296006    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:30.074895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:30.072474    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:30.814892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:30.805659    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:31.731890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:31.729848    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:32.161215 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.161101    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm" podUID="60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2" containerName="payload-pre-processing" containerID="cri-o://961e49feda722eb2f4b32938c5022dd3c6f7f0d2f3550c762e4345fb707e5d65" gracePeriod=25
Oct 31 13:33:32.456756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.456731    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"
Oct 31 13:33:32.602455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.602417    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2-kube-api-access-r8wh7\" (UniqueName: \"kubernetes.io/projected/60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2-kube-api-access-r8wh7\") pod \"60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2\" (UID: \"60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2\") "
Oct 31 13:33:32.602726 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.602706    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2-plugins-config-volume\") pod \"60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2\" (UID: \"60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2\") "
Oct 31 13:33:32.603185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.603160    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2-plugins-config-volume" pod "60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2" (UID: "60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:33:32.603596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.603575    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:33:32.604744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.604722    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2-kube-api-access-r8wh7" pod "60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2" (UID: "60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2"). InnerVolumeSpecName "kube-api-access-r8wh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:33:32.686962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.675161    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:32.706983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.706184    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8wh7\" (UniqueName: \"kubernetes.io/projected/60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2-kube-api-access-r8wh7\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:33:32.714902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.711796    4428 generic.go:356] "Generic (PLEG): container finished" podID="60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2" containerID="961e49feda722eb2f4b32938c5022dd3c6f7f0d2f3550c762e4345fb707e5d65" exitCode=0
Oct 31 13:33:32.714902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.712002    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm" event={"ID":"60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2","Type":"ContainerDied","Data":"961e49feda722eb2f4b32938c5022dd3c6f7f0d2f3550c762e4345fb707e5d65"}
Oct 31 13:33:32.714902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.712063    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm" event={"ID":"60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2","Type":"ContainerDied","Data":"cd2bef93767654a71c6d87e44c4c4102e6e4f54e7e32f3338ee42e774e671edd"}
Oct 31 13:33:32.714902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.712093    4428 scope.go:122] "RemoveContainer" containerID="961e49feda722eb2f4b32938c5022dd3c6f7f0d2f3550c762e4345fb707e5d65"
Oct 31 13:33:32.714902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.712268    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"
Oct 31 13:33:32.793628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.793594    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"]
Oct 31 13:33:32.794547 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.794523    4428 scope.go:122] "RemoveContainer" containerID="961e49feda722eb2f4b32938c5022dd3c6f7f0d2f3550c762e4345fb707e5d65"
Oct 31 13:33:32.794909 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:33:32.794887    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961e49feda722eb2f4b32938c5022dd3c6f7f0d2f3550c762e4345fb707e5d65\": container with ID starting with 961e49feda722eb2f4b32938c5022dd3c6f7f0d2f3550c762e4345fb707e5d65 not found: ID does not exist" containerID="961e49feda722eb2f4b32938c5022dd3c6f7f0d2f3550c762e4345fb707e5d65"
Oct 31 13:33:32.794999 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.794921    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961e49feda722eb2f4b32938c5022dd3c6f7f0d2f3550c762e4345fb707e5d65"} err="failed to get container status \"961e49feda722eb2f4b32938c5022dd3c6f7f0d2f3550c762e4345fb707e5d65\": rpc error: code = NotFound desc = could not find container \"961e49feda722eb2f4b32938c5022dd3c6f7f0d2f3550c762e4345fb707e5d65\": container with ID starting with 961e49feda722eb2f4b32938c5022dd3c6f7f0d2f3550c762e4345fb707e5d65 not found: ID does not exist"
Oct 31 13:33:32.794999 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:32.794941    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a8c1600c-764447d479-426vm"]
Oct 31 13:33:33.402666 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:33.401136    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:34.242000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:34.235781    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:34.756048 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:34.756007    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2" path="/var/lib/kubelet/pods/60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2/volumes"
Oct 31 13:33:35.102018 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:35.101537    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:35.699947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:35.699914    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:35.922893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:35.921393    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:36.228893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:36.228354    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:36.228893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:36.228427    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"]
Oct 31 13:33:36.232916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:36.232883    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a8c1600c-p5db7"]
Oct 31 13:33:36.419892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:36.417213    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:36.756539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:36.756505    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5176c0-4a26-4347-8bcb-8411e8c98f4f" path="/var/lib/kubelet/pods/ea5176c0-4a26-4347-8bcb-8411e8c98f4f/volumes"
Oct 31 13:33:36.769914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:36.769888    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:37.424893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:37.422278    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:38.354272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:38.354218    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:38.978480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:38.978454    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:39.176559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:39.176523    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:39.379487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:39.379395    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:39.616889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:39.612978    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:40.303997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:40.303960    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:41.141896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:41.137647    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:42.138626 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:42.138541    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:42.805178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:42.805144    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:42.971104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:42.971070    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:43.181953 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:43.180960    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:43.463127 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:43.463099    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:43.867282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:43.867207    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:44.016034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.015982    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"]
Oct 31 13:33:44.016678 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.016661    4428 cpu_manager.go:414] "containerMap: removing container" podUID="ea5176c0-4a26-4347-8bcb-8411e8c98f4f" containerName="revoke-keys"
Oct 31 13:33:44.016678 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.016679    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5176c0-4a26-4347-8bcb-8411e8c98f4f" containerName="revoke-keys"
Oct 31 13:33:44.016792 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.016725    4428 cpu_manager.go:414] "containerMap: removing container" podUID="60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2" containerName="payload-pre-processing"
Oct 31 13:33:44.016792 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.016730    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2" containerName="payload-pre-processing"
Oct 31 13:33:44.016858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.016832    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a4349a-7e10-4a5a-9ce4-c17f21f4b3c2" containerName="payload-pre-processing"
Oct 31 13:33:44.016858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.016844    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5176c0-4a26-4347-8bcb-8411e8c98f4f" containerName="revoke-keys"
Oct 31 13:33:44.022240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.022219    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"
Oct 31 13:33:44.031981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.031959    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"]
Oct 31 13:33:44.032725 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.032705    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-466002\""
Oct 31 13:33:44.040602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.040581    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-466002-dockercfg-xh64n\""
Oct 31 13:33:44.074414 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.074379    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"]
Oct 31 13:33:44.079988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.079916    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c80e96-fj8ld"]
Oct 31 13:33:44.124494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.124433    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzvj9\" (UniqueName: \"kubernetes.io/projected/d77bf054-b344-4ea7-b809-ea8ea990aef6-kube-api-access-qzvj9\") pod \"payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j\" (UID: \"d77bf054-b344-4ea7-b809-ea8ea990aef6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"
Oct 31 13:33:44.124607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.124497    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77bf054-b344-4ea7-b809-ea8ea990aef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j\" (UID: \"d77bf054-b344-4ea7-b809-ea8ea990aef6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"
Oct 31 13:33:44.185343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.185307    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:44.226532 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.226496    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77bf054-b344-4ea7-b809-ea8ea990aef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j\" (UID: \"d77bf054-b344-4ea7-b809-ea8ea990aef6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"
Oct 31 13:33:44.226859 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.226840    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzvj9\" (UniqueName: \"kubernetes.io/projected/d77bf054-b344-4ea7-b809-ea8ea990aef6-kube-api-access-qzvj9\") pod \"payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j\" (UID: \"d77bf054-b344-4ea7-b809-ea8ea990aef6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"
Oct 31 13:33:44.227357 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.227309    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77bf054-b344-4ea7-b809-ea8ea990aef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j\" (UID: \"d77bf054-b344-4ea7-b809-ea8ea990aef6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"
Oct 31 13:33:44.257481 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.257456    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzvj9\" (UniqueName: \"kubernetes.io/projected/d77bf054-b344-4ea7-b809-ea8ea990aef6-kube-api-access-qzvj9\") pod \"payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j\" (UID: \"d77bf054-b344-4ea7-b809-ea8ea990aef6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"
Oct 31 13:33:44.339741 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.339707    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"
Oct 31 13:33:44.531361 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.531328    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:44.917580 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.917545    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46cd4d85-81a9-44ae-a847-28cb5046e0d9" path="/var/lib/kubelet/pods/46cd4d85-81a9-44ae-a847-28cb5046e0d9/volumes"
Oct 31 13:33:44.948552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.945219    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"]
Oct 31 13:33:44.951235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.951208    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"
Oct 31 13:33:44.961713 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.961683    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"]
Oct 31 13:33:44.962491 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.962394    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:33:44.962491 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:44.962490    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:33:45.007858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:45.007837    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"]
Oct 31 13:33:45.011849 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:33:45.011828    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77bf054_b344_4ea7_b809_ea8ea990aef6.slice/crio-f08efe23c28a68c840e8ac743a58c1ac845af420329d9b053c05a017ba83fbfe WatchSource:0}: Error finding container f08efe23c28a68c840e8ac743a58c1ac845af420329d9b053c05a017ba83fbfe: Status 404 returned error can't find the container with id f08efe23c28a68c840e8ac743a58c1ac845af420329d9b053c05a017ba83fbfe
Oct 31 13:33:45.017462 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:45.017438    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf9ld\" (UniqueName: \"kubernetes.io/projected/10b20618-6142-430f-bfc5-108a93cb3b54-kube-api-access-cf9ld\") pod \"maas-api-revoke-sub-51f518f2-24nd7\" (UID: \"10b20618-6142-430f-bfc5-108a93cb3b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"
Oct 31 13:33:45.017638 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:45.017617    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b20618-6142-430f-bfc5-108a93cb3b54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51f518f2-24nd7\" (UID: \"10b20618-6142-430f-bfc5-108a93cb3b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"
Oct 31 13:33:45.017800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:45.017781    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b20618-6142-430f-bfc5-108a93cb3b54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51f518f2-24nd7\" (UID: \"10b20618-6142-430f-bfc5-108a93cb3b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"
Oct 31 13:33:45.119809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:45.119788    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf9ld\" (UniqueName: \"kubernetes.io/projected/10b20618-6142-430f-bfc5-108a93cb3b54-kube-api-access-cf9ld\") pod \"maas-api-revoke-sub-51f518f2-24nd7\" (UID: \"10b20618-6142-430f-bfc5-108a93cb3b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"
Oct 31 13:33:45.119939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:45.119924    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b20618-6142-430f-bfc5-108a93cb3b54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51f518f2-24nd7\" (UID: \"10b20618-6142-430f-bfc5-108a93cb3b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"
Oct 31 13:33:45.120085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:45.120068    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b20618-6142-430f-bfc5-108a93cb3b54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51f518f2-24nd7\" (UID: \"10b20618-6142-430f-bfc5-108a93cb3b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"
Oct 31 13:33:45.120803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:45.120769    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b20618-6142-430f-bfc5-108a93cb3b54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51f518f2-24nd7\" (UID: \"10b20618-6142-430f-bfc5-108a93cb3b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"
Oct 31 13:33:45.122526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:45.122507    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b20618-6142-430f-bfc5-108a93cb3b54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51f518f2-24nd7\" (UID: \"10b20618-6142-430f-bfc5-108a93cb3b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"
Oct 31 13:33:45.136312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:45.136292    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf9ld\" (UniqueName: \"kubernetes.io/projected/10b20618-6142-430f-bfc5-108a93cb3b54-kube-api-access-cf9ld\") pod \"maas-api-revoke-sub-51f518f2-24nd7\" (UID: \"10b20618-6142-430f-bfc5-108a93cb3b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"
Oct 31 13:33:45.280059 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:45.280023    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"
Oct 31 13:33:45.394887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:45.394570    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:45.908993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:45.898568    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"]
Oct 31 13:33:45.934832 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:33:45.934802    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b20618_6142_430f_bfc5_108a93cb3b54.slice/crio-c6fc0caca979d1a0dc85281ea8b33095538a57bf3512c5bd9e61b6ec0597c0d8 WatchSource:0}: Error finding container c6fc0caca979d1a0dc85281ea8b33095538a57bf3512c5bd9e61b6ec0597c0d8: Status 404 returned error can't find the container with id c6fc0caca979d1a0dc85281ea8b33095538a57bf3512c5bd9e61b6ec0597c0d8
Oct 31 13:33:45.937268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:45.937235    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j" event={"ID":"d77bf054-b344-4ea7-b809-ea8ea990aef6","Type":"ContainerStarted","Data":"d08fb901c5c3403583cf1587c95579da9b62d1361cb0991420fa07f614d97bd9"}
Oct 31 13:33:45.937379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:45.937288    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j" event={"ID":"d77bf054-b344-4ea7-b809-ea8ea990aef6","Type":"ContainerStarted","Data":"f08efe23c28a68c840e8ac743a58c1ac845af420329d9b053c05a017ba83fbfe"}
Oct 31 13:33:45.937379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:45.937346    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"
Oct 31 13:33:45.962566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:45.962519    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j" podStartSLOduration=2.962506306 podStartE2EDuration="2.962506306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:33:45.95795884 +0000 UTC m=+1823.356307462" watchObservedRunningTime="2026-10-31 13:33:45.962506306 +0000 UTC m=+1823.360854930"
Oct 31 13:33:46.301898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:46.277494    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:46.639951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:46.635096    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:46.832224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:46.832188    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:46.946702 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:46.946674    4428 generic.go:356] "Generic (PLEG): container finished" podID="10b20618-6142-430f-bfc5-108a93cb3b54" containerID="adb3e667aa9378dae670da1c4ce7fa9e31370ba4aeaad5d3dea9d58341b710dd" exitCode=0
Oct 31 13:33:46.947146 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:46.946756    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7" event={"ID":"10b20618-6142-430f-bfc5-108a93cb3b54","Type":"ContainerDied","Data":"adb3e667aa9378dae670da1c4ce7fa9e31370ba4aeaad5d3dea9d58341b710dd"}
Oct 31 13:33:46.947146 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:46.946828    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7" event={"ID":"10b20618-6142-430f-bfc5-108a93cb3b54","Type":"ContainerStarted","Data":"c6fc0caca979d1a0dc85281ea8b33095538a57bf3512c5bd9e61b6ec0597c0d8"}
Oct 31 13:33:47.006334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:47.006305    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:47.551368 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:47.551327    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:47.791891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:47.791585    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:48.052582 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:48.052544    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:48.203200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:48.203172    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"
Oct 31 13:33:48.247360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:48.247321    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10b20618-6142-430f-bfc5-108a93cb3b54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b20618-6142-430f-bfc5-108a93cb3b54-maas-api-xks-ca\") pod \"10b20618-6142-430f-bfc5-108a93cb3b54\" (UID: \"10b20618-6142-430f-bfc5-108a93cb3b54\") "
Oct 31 13:33:48.247522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:48.247481    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b20618-6142-430f-bfc5-108a93cb3b54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b20618-6142-430f-bfc5-108a93cb3b54-maas-api-openshift-ca\") pod \"10b20618-6142-430f-bfc5-108a93cb3b54\" (UID: \"10b20618-6142-430f-bfc5-108a93cb3b54\") "
Oct 31 13:33:48.247573 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:48.247546    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b20618-6142-430f-bfc5-108a93cb3b54-kube-api-access-cf9ld\" (UniqueName: \"kubernetes.io/projected/10b20618-6142-430f-bfc5-108a93cb3b54-kube-api-access-cf9ld\") pod \"10b20618-6142-430f-bfc5-108a93cb3b54\" (UID: \"10b20618-6142-430f-bfc5-108a93cb3b54\") "
Oct 31 13:33:48.247987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:48.247949    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b20618-6142-430f-bfc5-108a93cb3b54-maas-api-openshift-ca" pod "10b20618-6142-430f-bfc5-108a93cb3b54" (UID: "10b20618-6142-430f-bfc5-108a93cb3b54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:33:48.248662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:48.248632    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b20618-6142-430f-bfc5-108a93cb3b54-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:33:48.249790 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:48.249761    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10b20618-6142-430f-bfc5-108a93cb3b54-maas-api-xks-ca" pod "10b20618-6142-430f-bfc5-108a93cb3b54" (UID: "10b20618-6142-430f-bfc5-108a93cb3b54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:33:48.249994 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:48.249970    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b20618-6142-430f-bfc5-108a93cb3b54-kube-api-access-cf9ld" pod "10b20618-6142-430f-bfc5-108a93cb3b54" (UID: "10b20618-6142-430f-bfc5-108a93cb3b54"). InnerVolumeSpecName "kube-api-access-cf9ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:33:48.298400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:48.298368    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:48.350563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:48.350534    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b20618-6142-430f-bfc5-108a93cb3b54-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:33:48.350563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:48.350560    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf9ld\" (UniqueName: \"kubernetes.io/projected/10b20618-6142-430f-bfc5-108a93cb3b54-kube-api-access-cf9ld\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:33:48.740959 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:33:48.740917    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:33:49.023116 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.023037    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"
Oct 31 13:33:49.023116 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.023072    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7" event={"ID":"10b20618-6142-430f-bfc5-108a93cb3b54","Type":"ContainerDied","Data":"c6fc0caca979d1a0dc85281ea8b33095538a57bf3512c5bd9e61b6ec0597c0d8"}
Oct 31 13:33:49.023300 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.023120    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6fc0caca979d1a0dc85281ea8b33095538a57bf3512c5bd9e61b6ec0597c0d8"
Oct 31 13:33:49.054294 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.054132    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"]
Oct 31 13:33:49.059344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.059321    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a52ccf6-j6v49"]
Oct 31 13:33:49.191862 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.191828    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:49.347792 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.347720    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"]
Oct 31 13:33:49.348482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.348467    4428 cpu_manager.go:414] "containerMap: removing container" podUID="10b20618-6142-430f-bfc5-108a93cb3b54" containerName="revoke-keys"
Oct 31 13:33:49.348537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.348483    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b20618-6142-430f-bfc5-108a93cb3b54" containerName="revoke-keys"
Oct 31 13:33:49.348596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.348587    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b20618-6142-430f-bfc5-108a93cb3b54" containerName="revoke-keys"
Oct 31 13:33:49.352556 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.352529    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"
Oct 31 13:33:49.364197 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.364175    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-33c10a\""
Oct 31 13:33:49.367809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.367781    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkv59\" (UniqueName: \"kubernetes.io/projected/35f82843-6eb3-490b-8c6c-9dad7cce21a1-kube-api-access-kkv59\") pod \"payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb\" (UID: \"35f82843-6eb3-490b-8c6c-9dad7cce21a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"
Oct 31 13:33:49.367936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.367835    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f82843-6eb3-490b-8c6c-9dad7cce21a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb\" (UID: \"35f82843-6eb3-490b-8c6c-9dad7cce21a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"
Oct 31 13:33:49.377450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.377430    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"]
Oct 31 13:33:49.463200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.463162    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:49.494579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.470592    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkv59\" (UniqueName: \"kubernetes.io/projected/35f82843-6eb3-490b-8c6c-9dad7cce21a1-kube-api-access-kkv59\") pod \"payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb\" (UID: \"35f82843-6eb3-490b-8c6c-9dad7cce21a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"
Oct 31 13:33:49.495294 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.495259    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f82843-6eb3-490b-8c6c-9dad7cce21a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb\" (UID: \"35f82843-6eb3-490b-8c6c-9dad7cce21a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"
Oct 31 13:33:49.545889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.533746    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f82843-6eb3-490b-8c6c-9dad7cce21a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb\" (UID: \"35f82843-6eb3-490b-8c6c-9dad7cce21a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"
Oct 31 13:33:49.825881 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.819699    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkv59\" (UniqueName: \"kubernetes.io/projected/35f82843-6eb3-490b-8c6c-9dad7cce21a1-kube-api-access-kkv59\") pod \"payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb\" (UID: \"35f82843-6eb3-490b-8c6c-9dad7cce21a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"
Oct 31 13:33:49.968096 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:49.968060    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"
Oct 31 13:33:50.259704 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:33:50.259674    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f82843_6eb3_490b_8c6c_9dad7cce21a1.slice/crio-5932f5e768c4e9223895356f1b100fc182287a6e1f1d3a05c7dd136a8403eb78 WatchSource:0}: Error finding container 5932f5e768c4e9223895356f1b100fc182287a6e1f1d3a05c7dd136a8403eb78: Status 404 returned error can't find the container with id 5932f5e768c4e9223895356f1b100fc182287a6e1f1d3a05c7dd136a8403eb78
Oct 31 13:33:50.260057 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:50.259978    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"]
Oct 31 13:33:50.445892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:50.444847    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:50.757583 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:50.757548    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="096c7ece-20c0-409f-b7f9-384562f5f452" path="/var/lib/kubelet/pods/096c7ece-20c0-409f-b7f9-384562f5f452/volumes"
Oct 31 13:33:51.086699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:51.086601    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb" event={"ID":"35f82843-6eb3-490b-8c6c-9dad7cce21a1","Type":"ContainerStarted","Data":"88943beab6bab49dd7d59d59d160fd7334fab1952768aac0ffa52a5baa3d7ea8"}
Oct 31 13:33:51.086699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:51.086653    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb" event={"ID":"35f82843-6eb3-490b-8c6c-9dad7cce21a1","Type":"ContainerStarted","Data":"5932f5e768c4e9223895356f1b100fc182287a6e1f1d3a05c7dd136a8403eb78"}
Oct 31 13:33:51.086699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:51.086680    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"
Oct 31 13:33:51.112664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:51.112610    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb" podStartSLOduration=2.112596105 podStartE2EDuration="2.112596105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:33:51.111910261 +0000 UTC m=+1828.510258886" watchObservedRunningTime="2026-10-31 13:33:51.112596105 +0000 UTC m=+1828.510944784"
Oct 31 13:33:51.456948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:51.453104    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:52.210184 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:52.210152    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:53.008898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:53.006768    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:53.836035 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:53.835463    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:53.994908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:53.994133    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:54.833798 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:54.833764    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:55.058891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:55.054479    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:55.292092 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:55.292058    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:55.460856 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:55.460821    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:55.951977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:55.947500    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:56.139023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:56.137090    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:56.955988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:56.954616    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"
Oct 31 13:33:57.162130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:57.162087    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:58.007910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:58.007740    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:58.088012 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:58.087975    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"]
Oct 31 13:33:58.097894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:58.097442    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"]
Oct 31 13:33:58.107904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:58.106069    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76da4665-qbwf5"]
Oct 31 13:33:58.126897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:58.124931    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2302af36-fbbpr"]
Oct 31 13:33:58.756942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:58.756904    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa77e84-829b-49e8-8f79-8aebf5894b42" path="/var/lib/kubelet/pods/6aa77e84-829b-49e8-8f79-8aebf5894b42/volumes"
Oct 31 13:33:58.757363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:58.757350    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b31f49-3da5-4a8f-a226-42b79105c854" path="/var/lib/kubelet/pods/95b31f49-3da5-4a8f-a226-42b79105c854/volumes"
Oct 31 13:33:59.031091 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:59.031002    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:33:59.218226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:33:59.218196    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:00.086752 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.086722    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"]
Oct 31 13:34:00.095655 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.095629    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 13:34:00.100200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.100175    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-33c10a-serving-cert\""
Oct 31 13:34:00.148502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.148460    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"]
Oct 31 13:34:00.150406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.150370    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bdda64-b266-4073-b001-e8313083d4fa-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58\" (UID: \"31bdda64-b266-4073-b001-e8313083d4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 13:34:00.150545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.150475    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krzkl\" (UniqueName: \"kubernetes.io/projected/31bdda64-b266-4073-b001-e8313083d4fa-kube-api-access-krzkl\") pod \"maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58\" (UID: \"31bdda64-b266-4073-b001-e8313083d4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 13:34:00.150545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.150522    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31bdda64-b266-4073-b001-e8313083d4fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58\" (UID: \"31bdda64-b266-4073-b001-e8313083d4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 13:34:00.150672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.150646    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bdda64-b266-4073-b001-e8313083d4fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58\" (UID: \"31bdda64-b266-4073-b001-e8313083d4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 13:34:00.251657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.251619    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bdda64-b266-4073-b001-e8313083d4fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58\" (UID: \"31bdda64-b266-4073-b001-e8313083d4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 13:34:00.251822 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.251804    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bdda64-b266-4073-b001-e8313083d4fa-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58\" (UID: \"31bdda64-b266-4073-b001-e8313083d4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 13:34:00.251904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.251860    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krzkl\" (UniqueName: \"kubernetes.io/projected/31bdda64-b266-4073-b001-e8313083d4fa-kube-api-access-krzkl\") pod \"maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58\" (UID: \"31bdda64-b266-4073-b001-e8313083d4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 13:34:00.251950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.251906    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31bdda64-b266-4073-b001-e8313083d4fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58\" (UID: \"31bdda64-b266-4073-b001-e8313083d4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 13:34:00.252411 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.252386    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bdda64-b266-4073-b001-e8313083d4fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58\" (UID: \"31bdda64-b266-4073-b001-e8313083d4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 13:34:00.254637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.254614    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bdda64-b266-4073-b001-e8313083d4fa-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58\" (UID: \"31bdda64-b266-4073-b001-e8313083d4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 13:34:00.254747 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.254723    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31bdda64-b266-4073-b001-e8313083d4fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58\" (UID: \"31bdda64-b266-4073-b001-e8313083d4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 13:34:00.264984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.264956    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krzkl\" (UniqueName: \"kubernetes.io/projected/31bdda64-b266-4073-b001-e8313083d4fa-kube-api-access-krzkl\") pod \"maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58\" (UID: \"31bdda64-b266-4073-b001-e8313083d4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 13:34:00.320528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.320476    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:00.426317 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.426195    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 13:34:00.518706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:00.518670    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:01.440760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:01.440684    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:01.772461 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:01.751402    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"]
Oct 31 13:34:01.881425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:01.881384    4428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:34:02.120590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:02.120212    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"
Oct 31 13:34:02.452120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:02.445301    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:02.465470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:02.465421    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58" event={"ID":"31bdda64-b266-4073-b001-e8313083d4fa","Type":"ContainerStarted","Data":"49e1963ae566ef887704439c18a246af5cab8a1e3a64fcdd9966775819e1e19d"}
Oct 31 13:34:03.308174 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:03.308120    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:03.475652 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:03.475610    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58" event={"ID":"31bdda64-b266-4073-b001-e8313083d4fa","Type":"ContainerStarted","Data":"d2d92a1a5c8aadeb02847fa8a5588a213215113abd99cc188e4fc7b79fcd78cf"}
Oct 31 13:34:03.476055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:03.475699    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 13:34:03.501576 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:03.501533    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58" podStartSLOduration=3.060099311 podStartE2EDuration="3.501520907s" totalImagesPullingTime="441.421596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:03.497198971 +0000 UTC m=+1840.895547595" watchObservedRunningTime="2026-10-31 13:34:03.501520907 +0000 UTC m=+1840.899869570"
Oct 31 13:34:03.834487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:03.834434    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:04.155974 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:04.155456    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:04.921907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:04.920744    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:05.693017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:05.692839    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:06.444058 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:06.444009    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:07.421580 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:07.421498    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:08.031474 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:08.031438    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"]
Oct 31 13:34:08.036015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:08.035996    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da18ed15-djtsk"]
Oct 31 13:34:08.078356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:08.076441    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:08.398055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:08.397959    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:08.709097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:08.709004    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:08.759137 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:08.759103    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852864de-714b-4f59-98b9-e92da30271b1" path="/var/lib/kubelet/pods/852864de-714b-4f59-98b9-e92da30271b1/volumes"
Oct 31 13:34:09.042214 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.042185    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"]
Oct 31 13:34:09.047552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.047530    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.063966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.063912    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-752869-openshift-default-dockercfg-h2npg\""
Oct 31 13:34:09.067290 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.067237    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"]
Oct 31 13:34:09.138606 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.138573    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:09.156222 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.156192    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-envoy\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.156360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.156238    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istiod-ca-cert\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.156360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.156327    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtvv8\" (UniqueName: \"kubernetes.io/projected/d3179e22-1592-44a5-8193-8e7e9aee4b5e-kube-api-access-rtvv8\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.156461 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.156419    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-workload-certs\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.156461 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.156442    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-workload-socket\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.156604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.156583    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-token\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.156643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.156618    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-credential-socket\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.156695 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.156657    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-podinfo\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.156749 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.156736    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-data\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.258536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.258496    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-envoy\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.258723 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.258545    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istiod-ca-cert\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.258723 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.258585    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtvv8\" (UniqueName: \"kubernetes.io/projected/d3179e22-1592-44a5-8193-8e7e9aee4b5e-kube-api-access-rtvv8\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.258723 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.258657    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-workload-certs\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.258723 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.258683    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-workload-socket\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.258956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.258747    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-token\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.258956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.258779    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-credential-socket\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.258956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.258819    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-podinfo\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.258956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.258900    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-data\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.259273 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.259095    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-workload-certs\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.259273 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.259205    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-workload-socket\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.259273 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.259244    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-credential-socket\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.259437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.259339    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-data\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.259437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.259420    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istiod-ca-cert\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.261102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.261082    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-envoy\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.261553 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.261532    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-podinfo\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.291278 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.289893    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtvv8\" (UniqueName: \"kubernetes.io/projected/d3179e22-1592-44a5-8193-8e7e9aee4b5e-kube-api-access-rtvv8\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.310136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.309814    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-token\") pod \"e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") " pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.310136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.310019    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:09.377891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.373145    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:09.551915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:09.549642    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 13:34:10.016381 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:34:10.016351    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3179e22_1592_44a5_8193_8e7e9aee4b5e.slice/crio-82056bf2fd0e5acb98d23da210a4bf6523c8380b31ae07813efda9f923c97557 WatchSource:0}: Error finding container 82056bf2fd0e5acb98d23da210a4bf6523c8380b31ae07813efda9f923c97557: Status 404 returned error can't find the container with id 82056bf2fd0e5acb98d23da210a4bf6523c8380b31ae07813efda9f923c97557
Oct 31 13:34:10.016777 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:10.016751    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"]
Oct 31 13:34:10.019528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:10.019495    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:34:10.019611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:10.019566    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:34:10.019611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:10.019598    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:34:10.219546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:10.219518    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:10.500893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:10.499673    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:10.877945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:10.875611    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw" event={"ID":"d3179e22-1592-44a5-8193-8e7e9aee4b5e","Type":"ContainerStarted","Data":"cbb534617b29013f4c84e26ed1b7c2bf413c252d966860f002d75069f26b5e10"}
Oct 31 13:34:10.877945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:10.875684    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw" event={"ID":"d3179e22-1592-44a5-8193-8e7e9aee4b5e","Type":"ContainerStarted","Data":"82056bf2fd0e5acb98d23da210a4bf6523c8380b31ae07813efda9f923c97557"}
Oct 31 13:34:10.925919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:10.921955    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw" podStartSLOduration=1.921935092 podStartE2EDuration="1.921935092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:10.918528962 +0000 UTC m=+1848.316877588" watchObservedRunningTime="2026-10-31 13:34:10.921935092 +0000 UTC m=+1848.320283713"
Oct 31 13:34:11.360895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:11.354699    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:11.404898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:11.404127    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:11.431417 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:11.429165    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:11.908005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:11.907981    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:11.909302 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:11.909286    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:34:12.270259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:12.270220    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:13.100714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:13.100680    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:13.282079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:13.282000    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:13.456576 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:13.456493    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:13.625589 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:13.625527    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:14.492603 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:14.492572    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:14.874730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:14.874637    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:14.886408 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:14.886380    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"]
Oct 31 13:34:14.891038 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:14.891020    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"
Oct 31 13:34:14.896050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:14.896019    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-752869-dockercfg-62hx9\""
Oct 31 13:34:14.896325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:14.896301    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-752869\""
Oct 31 13:34:14.901929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:14.901714    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"]
Oct 31 13:34:14.944944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:14.944912    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"]
Oct 31 13:34:14.949596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:14.949570    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"
Oct 31 13:34:14.970023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:14.969991    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"]
Oct 31 13:34:15.137945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:15.104350    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:15.164892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:15.123150    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8baa85c2-ada3-42ca-849c-afffc3e1c13e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b\" (UID: \"8baa85c2-ada3-42ca-849c-afffc3e1c13e\") " pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"
Oct 31 13:34:15.164892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:15.150230    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxlxf\" (UniqueName: \"kubernetes.io/projected/9162cb66-9fcd-4fa4-89d2-f7044b8870a4-kube-api-access-sxlxf\") pod \"payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml\" (UID: \"9162cb66-9fcd-4fa4-89d2-f7044b8870a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"
Oct 31 13:34:15.164892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:15.150284    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwj9l\" (UniqueName: \"kubernetes.io/projected/8baa85c2-ada3-42ca-849c-afffc3e1c13e-kube-api-access-fwj9l\") pod \"payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b\" (UID: \"8baa85c2-ada3-42ca-849c-afffc3e1c13e\") " pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"
Oct 31 13:34:15.164892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:15.150387    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9162cb66-9fcd-4fa4-89d2-f7044b8870a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml\" (UID: \"9162cb66-9fcd-4fa4-89d2-f7044b8870a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"
Oct 31 13:34:15.373664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:15.373630    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxlxf\" (UniqueName: \"kubernetes.io/projected/9162cb66-9fcd-4fa4-89d2-f7044b8870a4-kube-api-access-sxlxf\") pod \"payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml\" (UID: \"9162cb66-9fcd-4fa4-89d2-f7044b8870a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"
Oct 31 13:34:15.374330 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:15.374289    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwj9l\" (UniqueName: \"kubernetes.io/projected/8baa85c2-ada3-42ca-849c-afffc3e1c13e-kube-api-access-fwj9l\") pod \"payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b\" (UID: \"8baa85c2-ada3-42ca-849c-afffc3e1c13e\") " pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"
Oct 31 13:34:15.374893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:15.374855    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9162cb66-9fcd-4fa4-89d2-f7044b8870a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml\" (UID: \"9162cb66-9fcd-4fa4-89d2-f7044b8870a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"
Oct 31 13:34:15.375320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:15.375298    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8baa85c2-ada3-42ca-849c-afffc3e1c13e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b\" (UID: \"8baa85c2-ada3-42ca-849c-afffc3e1c13e\") " pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"
Oct 31 13:34:15.376650 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:15.376627    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8baa85c2-ada3-42ca-849c-afffc3e1c13e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b\" (UID: \"8baa85c2-ada3-42ca-849c-afffc3e1c13e\") " pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"
Oct 31 13:34:15.398119 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:15.398047    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9162cb66-9fcd-4fa4-89d2-f7044b8870a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml\" (UID: \"9162cb66-9fcd-4fa4-89d2-f7044b8870a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"
Oct 31 13:34:15.439931 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:15.432351    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxlxf\" (UniqueName: \"kubernetes.io/projected/9162cb66-9fcd-4fa4-89d2-f7044b8870a4-kube-api-access-sxlxf\") pod \"payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml\" (UID: \"9162cb66-9fcd-4fa4-89d2-f7044b8870a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"
Oct 31 13:34:15.482895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:15.478000    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwj9l\" (UniqueName: \"kubernetes.io/projected/8baa85c2-ada3-42ca-849c-afffc3e1c13e-kube-api-access-fwj9l\") pod \"payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b\" (UID: \"8baa85c2-ada3-42ca-849c-afffc3e1c13e\") " pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"
Oct 31 13:34:15.514720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:15.514687    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"
Oct 31 13:34:15.573581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:15.573319    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"
Oct 31 13:34:15.733982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:15.727075    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:15.962288 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:34:15.962259    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8baa85c2_ada3_42ca_849c_afffc3e1c13e.slice/crio-7656b23a0dfee736c3ec4bfc930874ab28568588d344186df8ac4c71e5877d49 WatchSource:0}: Error finding container 7656b23a0dfee736c3ec4bfc930874ab28568588d344186df8ac4c71e5877d49: Status 404 returned error can't find the container with id 7656b23a0dfee736c3ec4bfc930874ab28568588d344186df8ac4c71e5877d49
Oct 31 13:34:15.963044 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:15.963018    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"]
Oct 31 13:34:16.013946 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:34:16.013918    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9162cb66_9fcd_4fa4_89d2_f7044b8870a4.slice/crio-397fc871cea58717cad81f0558f3ea3aedb30656e2d8e7427830f3526b0f4680 WatchSource:0}: Error finding container 397fc871cea58717cad81f0558f3ea3aedb30656e2d8e7427830f3526b0f4680: Status 404 returned error can't find the container with id 397fc871cea58717cad81f0558f3ea3aedb30656e2d8e7427830f3526b0f4680
Oct 31 13:34:16.015564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.015543    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"]
Oct 31 13:34:16.074885 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.074844    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"]
Oct 31 13:34:16.081088 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.081069    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"
Oct 31 13:34:16.087968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.087856    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:34:16.088980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.088943    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:34:16.089106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.089091    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"]
Oct 31 13:34:16.161534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.161504    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8c49\" (UniqueName: \"kubernetes.io/projected/fba5687b-c871-4a60-9c57-42170ccb5c7d-kube-api-access-m8c49\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs\" (UID: \"fba5687b-c871-4a60-9c57-42170ccb5c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"
Oct 31 13:34:16.161690 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.161673    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba5687b-c871-4a60-9c57-42170ccb5c7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs\" (UID: \"fba5687b-c871-4a60-9c57-42170ccb5c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"
Oct 31 13:34:16.161830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.161815    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba5687b-c871-4a60-9c57-42170ccb5c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs\" (UID: \"fba5687b-c871-4a60-9c57-42170ccb5c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"
Oct 31 13:34:16.263417 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.263332    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba5687b-c871-4a60-9c57-42170ccb5c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs\" (UID: \"fba5687b-c871-4a60-9c57-42170ccb5c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"
Oct 31 13:34:16.263589 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.263535    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8c49\" (UniqueName: \"kubernetes.io/projected/fba5687b-c871-4a60-9c57-42170ccb5c7d-kube-api-access-m8c49\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs\" (UID: \"fba5687b-c871-4a60-9c57-42170ccb5c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"
Oct 31 13:34:16.263706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.263691    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba5687b-c871-4a60-9c57-42170ccb5c7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs\" (UID: \"fba5687b-c871-4a60-9c57-42170ccb5c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"
Oct 31 13:34:16.264119 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.264101    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba5687b-c871-4a60-9c57-42170ccb5c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs\" (UID: \"fba5687b-c871-4a60-9c57-42170ccb5c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"
Oct 31 13:34:16.266413 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.266386    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba5687b-c871-4a60-9c57-42170ccb5c7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs\" (UID: \"fba5687b-c871-4a60-9c57-42170ccb5c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"
Oct 31 13:34:16.286355 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.286332    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8c49\" (UniqueName: \"kubernetes.io/projected/fba5687b-c871-4a60-9c57-42170ccb5c7d-kube-api-access-m8c49\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs\" (UID: \"fba5687b-c871-4a60-9c57-42170ccb5c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"
Oct 31 13:34:16.329559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.329530    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:16.399673 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.399650    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"
Oct 31 13:34:16.578367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.578296    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:16.785223 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:34:16.784348    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba5687b_c871_4a60_9c57_42170ccb5c7d.slice/crio-e3f1bfccc1989f84250a079dc912c0a6cd45b9d7dac9d71be742c763e74a6b99 WatchSource:0}: Error finding container e3f1bfccc1989f84250a079dc912c0a6cd45b9d7dac9d71be742c763e74a6b99: Status 404 returned error can't find the container with id e3f1bfccc1989f84250a079dc912c0a6cd45b9d7dac9d71be742c763e74a6b99
Oct 31 13:34:16.792603 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.792580    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"]
Oct 31 13:34:16.863467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.863443    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:16.953880 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.953840    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b" event={"ID":"8baa85c2-ada3-42ca-849c-afffc3e1c13e","Type":"ContainerStarted","Data":"976014cb385c6e29d29f15c4ef4ccc130e285dd7760e134bbe835b2b582e1ff4"}
Oct 31 13:34:16.954005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.953912    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b" event={"ID":"8baa85c2-ada3-42ca-849c-afffc3e1c13e","Type":"ContainerStarted","Data":"7656b23a0dfee736c3ec4bfc930874ab28568588d344186df8ac4c71e5877d49"}
Oct 31 13:34:16.954005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.953941    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"
Oct 31 13:34:16.955579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.955552    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml" event={"ID":"9162cb66-9fcd-4fa4-89d2-f7044b8870a4","Type":"ContainerStarted","Data":"f9dabf4f694b7e704015f5086384a9a623c2a0c749c72a6ad58c181011ab947b"}
Oct 31 13:34:16.955700 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.955604    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"
Oct 31 13:34:16.955700 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.955624    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml" event={"ID":"9162cb66-9fcd-4fa4-89d2-f7044b8870a4","Type":"ContainerStarted","Data":"397fc871cea58717cad81f0558f3ea3aedb30656e2d8e7427830f3526b0f4680"}
Oct 31 13:34:16.956860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.956839    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs" event={"ID":"fba5687b-c871-4a60-9c57-42170ccb5c7d","Type":"ContainerStarted","Data":"e3f1bfccc1989f84250a079dc912c0a6cd45b9d7dac9d71be742c763e74a6b99"}
Oct 31 13:34:16.979641 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.979601    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b" podStartSLOduration=2.979589309 podStartE2EDuration="2.979589309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:16.9753931 +0000 UTC m=+1854.373741724" watchObservedRunningTime="2026-10-31 13:34:16.979589309 +0000 UTC m=+1854.377937965"
Oct 31 13:34:16.996919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:16.996855    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml" podStartSLOduration=2.996838604 podStartE2EDuration="2.996838604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:16.994245429 +0000 UTC m=+1854.392594056" watchObservedRunningTime="2026-10-31 13:34:16.996838604 +0000 UTC m=+1854.395187229"
Oct 31 13:34:17.489540 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:17.489507    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:17.657269 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:17.657228    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:17.933680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:17.932475    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:17.985895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:17.981680    4428 generic.go:356] "Generic (PLEG): container finished" podID="fba5687b-c871-4a60-9c57-42170ccb5c7d" containerID="66c30904050b44452fd6c759f13ba29063c5cfa15dce940448579aadb0df6320" exitCode=0
Oct 31 13:34:17.985895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:17.982120    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs" event={"ID":"fba5687b-c871-4a60-9c57-42170ccb5c7d","Type":"ContainerDied","Data":"66c30904050b44452fd6c759f13ba29063c5cfa15dce940448579aadb0df6320"}
Oct 31 13:34:18.227893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:18.215585    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:18.646798 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:18.646720    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:18.742236 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:34:18.742194    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:34:18.923794 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:18.923690    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:19.054398 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.047573    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"]
Oct 31 13:34:19.056921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.056819    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a616a32-xmx68"]
Oct 31 13:34:19.104564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.104527    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"]
Oct 31 13:34:19.110584 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.110535    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"]
Oct 31 13:34:19.112140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.112114    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.118515 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.117913    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-abdcb1-openshift-default-dockercfg-jnc5d\""
Oct 31 13:34:19.132629 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.130496    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"]
Oct 31 13:34:19.132629 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.130614    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"
Oct 31 13:34:19.134632 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.133729    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"]
Oct 31 13:34:19.275896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.224444    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:19.313763 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.313721    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-podinfo\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.314028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.314002    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad597eab-3ce1-44a8-8e37-cd64c3c83657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-636e4838-sdprf\" (UID: \"ad597eab-3ce1-44a8-8e37-cd64c3c83657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"
Oct 31 13:34:19.314203 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.314187    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb4rp\" (UniqueName: \"kubernetes.io/projected/ad597eab-3ce1-44a8-8e37-cd64c3c83657-kube-api-access-lb4rp\") pod \"maas-api-revoke-sub-636e4838-sdprf\" (UID: \"ad597eab-3ce1-44a8-8e37-cd64c3c83657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"
Oct 31 13:34:19.314325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.314309    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-workload-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.314445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.314431    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5g9w\" (UniqueName: \"kubernetes.io/projected/b9fb27a2-762a-4256-8c53-89a6bb50e654-kube-api-access-f5g9w\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.314655 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.314624    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-workload-certs\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.314816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.314803    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad597eab-3ce1-44a8-8e37-cd64c3c83657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-636e4838-sdprf\" (UID: \"ad597eab-3ce1-44a8-8e37-cd64c3c83657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"
Oct 31 13:34:19.315122 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.315102    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-envoy\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.315316 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.315298    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-credential-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.315443 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.315429    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-data\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.315670 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.315654    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-token\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.315814 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.315798    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9fb27a2-762a-4256-8c53-89a6bb50e654-istiod-ca-cert\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.542281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.532403    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad597eab-3ce1-44a8-8e37-cd64c3c83657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-636e4838-sdprf\" (UID: \"ad597eab-3ce1-44a8-8e37-cd64c3c83657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"
Oct 31 13:34:19.542281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.532621    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-envoy\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.542281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.532663    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-credential-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.542281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.532720    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-data\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.542281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.532857    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-token\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.542281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.532934    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9fb27a2-762a-4256-8c53-89a6bb50e654-istiod-ca-cert\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.542281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.532974    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-podinfo\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.542281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.533001    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad597eab-3ce1-44a8-8e37-cd64c3c83657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-636e4838-sdprf\" (UID: \"ad597eab-3ce1-44a8-8e37-cd64c3c83657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"
Oct 31 13:34:19.542281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.533060    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb4rp\" (UniqueName: \"kubernetes.io/projected/ad597eab-3ce1-44a8-8e37-cd64c3c83657-kube-api-access-lb4rp\") pod \"maas-api-revoke-sub-636e4838-sdprf\" (UID: \"ad597eab-3ce1-44a8-8e37-cd64c3c83657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"
Oct 31 13:34:19.542281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.533100    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-workload-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.542281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.533144    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5g9w\" (UniqueName: \"kubernetes.io/projected/b9fb27a2-762a-4256-8c53-89a6bb50e654-kube-api-access-f5g9w\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.542281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.533241    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-workload-certs\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.542281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.533836    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-workload-certs\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.542281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.534696    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad597eab-3ce1-44a8-8e37-cd64c3c83657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-636e4838-sdprf\" (UID: \"ad597eab-3ce1-44a8-8e37-cd64c3c83657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"
Oct 31 13:34:19.616739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.585805    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-credential-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.616739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.595258    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-data\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.616739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.595722    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-workload-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.616739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.596622    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9fb27a2-762a-4256-8c53-89a6bb50e654-istiod-ca-cert\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.669891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.651713    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-envoy\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.837003 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.836970    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-token\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.839762 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.839730    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:19.851763 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.851732    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad597eab-3ce1-44a8-8e37-cd64c3c83657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-636e4838-sdprf\" (UID: \"ad597eab-3ce1-44a8-8e37-cd64c3c83657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"
Oct 31 13:34:19.852532 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.852495    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-podinfo\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.853314 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.853293    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb4rp\" (UniqueName: \"kubernetes.io/projected/ad597eab-3ce1-44a8-8e37-cd64c3c83657-kube-api-access-lb4rp\") pod \"maas-api-revoke-sub-636e4838-sdprf\" (UID: \"ad597eab-3ce1-44a8-8e37-cd64c3c83657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"
Oct 31 13:34:19.867971 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.866803    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5g9w\" (UniqueName: \"kubernetes.io/projected/b9fb27a2-762a-4256-8c53-89a6bb50e654-kube-api-access-f5g9w\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:19.893750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.893378    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"
Oct 31 13:34:19.912060 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.911965    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba5687b-c871-4a60-9c57-42170ccb5c7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba5687b-c871-4a60-9c57-42170ccb5c7d-maas-api-openshift-ca\") pod \"fba5687b-c871-4a60-9c57-42170ccb5c7d\" (UID: \"fba5687b-c871-4a60-9c57-42170ccb5c7d\") "
Oct 31 13:34:19.912247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.912161    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fba5687b-c871-4a60-9c57-42170ccb5c7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba5687b-c871-4a60-9c57-42170ccb5c7d-maas-api-xks-ca\") pod \"fba5687b-c871-4a60-9c57-42170ccb5c7d\" (UID: \"fba5687b-c871-4a60-9c57-42170ccb5c7d\") "
Oct 31 13:34:19.912247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.912231    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba5687b-c871-4a60-9c57-42170ccb5c7d-kube-api-access-m8c49\" (UniqueName: \"kubernetes.io/projected/fba5687b-c871-4a60-9c57-42170ccb5c7d-kube-api-access-m8c49\") pod \"fba5687b-c871-4a60-9c57-42170ccb5c7d\" (UID: \"fba5687b-c871-4a60-9c57-42170ccb5c7d\") "
Oct 31 13:34:19.913285 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.913249    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba5687b-c871-4a60-9c57-42170ccb5c7d-maas-api-openshift-ca" pod "fba5687b-c871-4a60-9c57-42170ccb5c7d" (UID: "fba5687b-c871-4a60-9c57-42170ccb5c7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:34:19.915233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.915208    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fba5687b-c871-4a60-9c57-42170ccb5c7d-maas-api-xks-ca" pod "fba5687b-c871-4a60-9c57-42170ccb5c7d" (UID: "fba5687b-c871-4a60-9c57-42170ccb5c7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:34:19.915532 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:19.915517    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba5687b-c871-4a60-9c57-42170ccb5c7d-kube-api-access-m8c49" pod "fba5687b-c871-4a60-9c57-42170ccb5c7d" (UID: "fba5687b-c871-4a60-9c57-42170ccb5c7d"). InnerVolumeSpecName "kube-api-access-m8c49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:34:20.040445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:20.017716    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba5687b-c871-4a60-9c57-42170ccb5c7d-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:34:20.040445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:20.017774    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba5687b-c871-4a60-9c57-42170ccb5c7d-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:34:20.040445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:20.017792    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8c49\" (UniqueName: \"kubernetes.io/projected/fba5687b-c871-4a60-9c57-42170ccb5c7d-kube-api-access-m8c49\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:34:20.133950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:20.064404    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:20.133950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:20.098235    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"
Oct 31 13:34:20.133950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:20.100188    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:20.184475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:20.183681    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs" event={"ID":"fba5687b-c871-4a60-9c57-42170ccb5c7d","Type":"ContainerDied","Data":"e3f1bfccc1989f84250a079dc912c0a6cd45b9d7dac9d71be742c763e74a6b99"}
Oct 31 13:34:20.184475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:20.183763    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"
Oct 31 13:34:20.184475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:20.183781    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3f1bfccc1989f84250a079dc912c0a6cd45b9d7dac9d71be742c763e74a6b99"
Oct 31 13:34:20.492712 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:34:20.492680    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9fb27a2_762a_4256_8c53_89a6bb50e654.slice/crio-99842e435721c810b37850cf12b390af15f306736013f26e9a74e930a78270ca WatchSource:0}: Error finding container 99842e435721c810b37850cf12b390af15f306736013f26e9a74e930a78270ca: Status 404 returned error can't find the container with id 99842e435721c810b37850cf12b390af15f306736013f26e9a74e930a78270ca
Oct 31 13:34:20.493605 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:34:20.493577    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad597eab_3ce1_44a8_8e37_cd64c3c83657.slice/crio-1cc4ad4e7cf32afd01e11edb39853faeb457c464e419438d63a1891f335461ef WatchSource:0}: Error finding container 1cc4ad4e7cf32afd01e11edb39853faeb457c464e419438d63a1891f335461ef: Status 404 returned error can't find the container with id 1cc4ad4e7cf32afd01e11edb39853faeb457c464e419438d63a1891f335461ef
Oct 31 13:34:20.493689 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:20.493666    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"]
Oct 31 13:34:20.495947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:20.495917    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:34:20.496019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:20.495977    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:34:20.496019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:20.496009    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:34:20.499813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:20.499789    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"]
Oct 31 13:34:20.759342 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:20.759262    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae32076-375d-48d2-8640-4e9ee3913caf" path="/var/lib/kubelet/pods/6ae32076-375d-48d2-8640-4e9ee3913caf/volumes"
Oct 31 13:34:20.901893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:20.895606    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:21.194204 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:21.194114    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n" event={"ID":"b9fb27a2-762a-4256-8c53-89a6bb50e654","Type":"ContainerStarted","Data":"8b2d78d52433b2821008621c03483af3a2af47d779c95e2c2a80f872e1a77d0e"}
Oct 31 13:34:21.194204 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:21.194168    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n" event={"ID":"b9fb27a2-762a-4256-8c53-89a6bb50e654","Type":"ContainerStarted","Data":"99842e435721c810b37850cf12b390af15f306736013f26e9a74e930a78270ca"}
Oct 31 13:34:21.196229 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:21.196201    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf" event={"ID":"ad597eab-3ce1-44a8-8e37-cd64c3c83657","Type":"ContainerStarted","Data":"1e2b378710a40083e48f83f8dcdaa4ded3a77ab8c6fc5ffa5d5ef77fff2e7a0c"}
Oct 31 13:34:21.196358 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:21.196246    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf" event={"ID":"ad597eab-3ce1-44a8-8e37-cd64c3c83657","Type":"ContainerStarted","Data":"1cc4ad4e7cf32afd01e11edb39853faeb457c464e419438d63a1891f335461ef"}
Oct 31 13:34:21.224631 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:21.224578    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n" podStartSLOduration=2.224560892 podStartE2EDuration="2.224560892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:21.220744722 +0000 UTC m=+1858.619093346" watchObservedRunningTime="2026-10-31 13:34:21.224560892 +0000 UTC m=+1858.622909518"
Oct 31 13:34:21.248450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:21.248404    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf" podStartSLOduration=2.248390116 podStartE2EDuration="2.248390116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:21.245832881 +0000 UTC m=+1858.644181505" watchObservedRunningTime="2026-10-31 13:34:21.248390116 +0000 UTC m=+1858.646738737"
Oct 31 13:34:21.792276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:21.792185    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:22.076851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:22.076822    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:22.082911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:22.082886    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:22.207076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:22.207044    4428 generic.go:356] "Generic (PLEG): container finished" podID="ad597eab-3ce1-44a8-8e37-cd64c3c83657" containerID="1e2b378710a40083e48f83f8dcdaa4ded3a77ab8c6fc5ffa5d5ef77fff2e7a0c" exitCode=0
Oct 31 13:34:22.207270 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:22.207130    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf" event={"ID":"ad597eab-3ce1-44a8-8e37-cd64c3c83657","Type":"ContainerDied","Data":"1e2b378710a40083e48f83f8dcdaa4ded3a77ab8c6fc5ffa5d5ef77fff2e7a0c"}
Oct 31 13:34:22.207537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:22.207519    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:22.208400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:22.208383    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:34:23.402983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:23.402948    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"
Oct 31 13:34:23.526292 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:23.526221    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:23.532973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:23.532943    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad597eab-3ce1-44a8-8e37-cd64c3c83657-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad597eab-3ce1-44a8-8e37-cd64c3c83657-maas-api-xks-ca\") pod \"ad597eab-3ce1-44a8-8e37-cd64c3c83657\" (UID: \"ad597eab-3ce1-44a8-8e37-cd64c3c83657\") "
Oct 31 13:34:23.533243 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:23.533201    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad597eab-3ce1-44a8-8e37-cd64c3c83657-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad597eab-3ce1-44a8-8e37-cd64c3c83657-maas-api-openshift-ca\") pod \"ad597eab-3ce1-44a8-8e37-cd64c3c83657\" (UID: \"ad597eab-3ce1-44a8-8e37-cd64c3c83657\") "
Oct 31 13:34:23.533443 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:23.533415    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad597eab-3ce1-44a8-8e37-cd64c3c83657-kube-api-access-lb4rp\" (UniqueName: \"kubernetes.io/projected/ad597eab-3ce1-44a8-8e37-cd64c3c83657-kube-api-access-lb4rp\") pod \"ad597eab-3ce1-44a8-8e37-cd64c3c83657\" (UID: \"ad597eab-3ce1-44a8-8e37-cd64c3c83657\") "
Oct 31 13:34:23.533843 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:23.533812    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad597eab-3ce1-44a8-8e37-cd64c3c83657-maas-api-openshift-ca" pod "ad597eab-3ce1-44a8-8e37-cd64c3c83657" (UID: "ad597eab-3ce1-44a8-8e37-cd64c3c83657"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:34:23.534309 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:23.534223    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad597eab-3ce1-44a8-8e37-cd64c3c83657-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:34:23.536013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:23.535849    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad597eab-3ce1-44a8-8e37-cd64c3c83657-maas-api-xks-ca" pod "ad597eab-3ce1-44a8-8e37-cd64c3c83657" (UID: "ad597eab-3ce1-44a8-8e37-cd64c3c83657"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:34:23.536431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:23.536407    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad597eab-3ce1-44a8-8e37-cd64c3c83657-kube-api-access-lb4rp" pod "ad597eab-3ce1-44a8-8e37-cd64c3c83657" (UID: "ad597eab-3ce1-44a8-8e37-cd64c3c83657"). InnerVolumeSpecName "kube-api-access-lb4rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:34:23.635988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:23.635960    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb4rp\" (UniqueName: \"kubernetes.io/projected/ad597eab-3ce1-44a8-8e37-cd64c3c83657-kube-api-access-lb4rp\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:34:23.635988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:23.635985    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad597eab-3ce1-44a8-8e37-cd64c3c83657-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:34:23.686478 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:23.686439    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:24.225643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:24.225620    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"
Oct 31 13:34:24.225836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:24.225670    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf" event={"ID":"ad597eab-3ce1-44a8-8e37-cd64c3c83657","Type":"ContainerDied","Data":"1cc4ad4e7cf32afd01e11edb39853faeb457c464e419438d63a1891f335461ef"}
Oct 31 13:34:24.225836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:24.225718    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cc4ad4e7cf32afd01e11edb39853faeb457c464e419438d63a1891f335461ef"
Oct 31 13:34:24.685906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:24.685718    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:24.864163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:24.864119    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:24.887998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:24.887963    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"]
Oct 31 13:34:24.890050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:24.890029    4428 cpu_manager.go:414] "containerMap: removing container" podUID="fba5687b-c871-4a60-9c57-42170ccb5c7d" containerName="revoke-keys"
Oct 31 13:34:24.890201 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:24.890181    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba5687b-c871-4a60-9c57-42170ccb5c7d" containerName="revoke-keys"
Oct 31 13:34:24.890286 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:24.890266    4428 cpu_manager.go:414] "containerMap: removing container" podUID="ad597eab-3ce1-44a8-8e37-cd64c3c83657" containerName="revoke-keys"
Oct 31 13:34:24.890286 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:24.890283    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad597eab-3ce1-44a8-8e37-cd64c3c83657" containerName="revoke-keys"
Oct 31 13:34:24.890662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:24.890644    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba5687b-c871-4a60-9c57-42170ccb5c7d" containerName="revoke-keys"
Oct 31 13:34:24.890726 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:24.890673    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad597eab-3ce1-44a8-8e37-cd64c3c83657" containerName="revoke-keys"
Oct 31 13:34:24.895892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:24.895856    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"]
Oct 31 13:34:24.896053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:24.896000    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"
Oct 31 13:34:24.910336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:24.910314    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-abdcb1\""
Oct 31 13:34:24.910336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:24.910319    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-abdcb1-dockercfg-d8qwm\""
Oct 31 13:34:25.130896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:25.130309    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8641866-a4d3-4c0d-8d8a-dd25e8572139-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5\" (UID: \"e8641866-a4d3-4c0d-8d8a-dd25e8572139\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"
Oct 31 13:34:25.130896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:25.130459    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zdmx\" (UniqueName: \"kubernetes.io/projected/e8641866-a4d3-4c0d-8d8a-dd25e8572139-kube-api-access-6zdmx\") pod \"payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5\" (UID: \"e8641866-a4d3-4c0d-8d8a-dd25e8572139\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"
Oct 31 13:34:25.272894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:25.272537    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8641866-a4d3-4c0d-8d8a-dd25e8572139-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5\" (UID: \"e8641866-a4d3-4c0d-8d8a-dd25e8572139\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"
Oct 31 13:34:25.272894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:25.272677    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zdmx\" (UniqueName: \"kubernetes.io/projected/e8641866-a4d3-4c0d-8d8a-dd25e8572139-kube-api-access-6zdmx\") pod \"payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5\" (UID: \"e8641866-a4d3-4c0d-8d8a-dd25e8572139\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"
Oct 31 13:34:25.275168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:25.274209    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8641866-a4d3-4c0d-8d8a-dd25e8572139-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5\" (UID: \"e8641866-a4d3-4c0d-8d8a-dd25e8572139\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"
Oct 31 13:34:25.308938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:25.308905    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zdmx\" (UniqueName: \"kubernetes.io/projected/e8641866-a4d3-4c0d-8d8a-dd25e8572139-kube-api-access-6zdmx\") pod \"payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5\" (UID: \"e8641866-a4d3-4c0d-8d8a-dd25e8572139\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"
Oct 31 13:34:25.531893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:25.529825    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"
Oct 31 13:34:25.597923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:25.589900    4428 scope.go:122] "RemoveContainer" containerID="e93ced3ac204493d9fc19d879700b414ec5329d250349a22e90fb41b52568a9a"
Oct 31 13:34:25.784950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:25.781569    4428 scope.go:122] "RemoveContainer" containerID="36f45770657d8925f2985f2ac47a64bcc37918564316e767dc0aee131033f039"
Oct 31 13:34:25.784950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:25.782634    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:25.864018 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:25.863979    4428 scope.go:122] "RemoveContainer" containerID="6633f9fbfd0ce91bb86582998aad1a3b968ff55425353561cea50fff324318e6"
Oct 31 13:34:26.317525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:26.317496    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:26.339727 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:34:26.339695    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8641866_a4d3_4c0d_8d8a_dd25e8572139.slice/crio-e2e8822e98cc608d923e9258d37fc43769fa287834ebff8cbec959c377b3081c WatchSource:0}: Error finding container e2e8822e98cc608d923e9258d37fc43769fa287834ebff8cbec959c377b3081c: Status 404 returned error can't find the container with id e2e8822e98cc608d923e9258d37fc43769fa287834ebff8cbec959c377b3081c
Oct 31 13:34:26.340486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:26.340464    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"]
Oct 31 13:34:26.489141 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:26.489097    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:26.890559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:26.888232    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:27.066889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:27.066834    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:27.344707 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:27.344670    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5" event={"ID":"e8641866-a4d3-4c0d-8d8a-dd25e8572139","Type":"ContainerStarted","Data":"0714db78f0b3bef9c995c8f3310afdf593508b279d76303f783d5246189760d5"}
Oct 31 13:34:27.345029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:27.345008    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5" event={"ID":"e8641866-a4d3-4c0d-8d8a-dd25e8572139","Type":"ContainerStarted","Data":"e2e8822e98cc608d923e9258d37fc43769fa287834ebff8cbec959c377b3081c"}
Oct 31 13:34:27.346419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:27.346396    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"
Oct 31 13:34:27.378245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:27.378184    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5" podStartSLOduration=3.378162116 podStartE2EDuration="3.378162116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:27.376306625 +0000 UTC m=+1864.774655250" watchObservedRunningTime="2026-10-31 13:34:27.378162116 +0000 UTC m=+1864.776510746"
Oct 31 13:34:27.905235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:27.905197    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:28.006341 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:28.006310    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"
Oct 31 13:34:28.012126 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:28.007004    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"
Oct 31 13:34:28.939679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:28.937730    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:29.465402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:29.465338    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:29.619264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:29.619230    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:29.822944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:29.814371    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:30.184240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:30.184147    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:30.731897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:30.728534    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:30.906890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:30.900123    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:31.801897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:31.797821    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:32.666893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:32.662687    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:33.579905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:33.579062    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:34.299529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:34.299496    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:34.503897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:34.502186    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:34.858892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:34.858205    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:35.887948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:35.887911    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:36.060916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:36.060766    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:36.357959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:36.307928    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"]
Oct 31 13:34:36.359301 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:36.359253    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:36.385827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:36.382234    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"
Oct 31 13:34:36.387495 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:36.387175    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cce3e7\""
Oct 31 13:34:36.397033 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:36.397005    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cce3e7-dockercfg-ncffd\""
Oct 31 13:34:36.424974 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:36.424945    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"]
Oct 31 13:34:36.709946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:36.709759    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnrvq\" (UniqueName: \"kubernetes.io/projected/e34f0c7f-398d-4614-8851-913948829895-kube-api-access-qnrvq\") pod \"payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9\" (UID: \"e34f0c7f-398d-4614-8851-913948829895\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"
Oct 31 13:34:36.732388 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:36.732347    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34f0c7f-398d-4614-8851-913948829895-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9\" (UID: \"e34f0c7f-398d-4614-8851-913948829895\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"
Oct 31 13:34:36.837365 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:36.837329    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnrvq\" (UniqueName: \"kubernetes.io/projected/e34f0c7f-398d-4614-8851-913948829895-kube-api-access-qnrvq\") pod \"payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9\" (UID: \"e34f0c7f-398d-4614-8851-913948829895\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"
Oct 31 13:34:36.840150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:36.838028    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34f0c7f-398d-4614-8851-913948829895-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9\" (UID: \"e34f0c7f-398d-4614-8851-913948829895\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"
Oct 31 13:34:36.843480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:36.841432    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34f0c7f-398d-4614-8851-913948829895-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9\" (UID: \"e34f0c7f-398d-4614-8851-913948829895\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"
Oct 31 13:34:36.986128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:36.985777    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnrvq\" (UniqueName: \"kubernetes.io/projected/e34f0c7f-398d-4614-8851-913948829895-kube-api-access-qnrvq\") pod \"payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9\" (UID: \"e34f0c7f-398d-4614-8851-913948829895\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"
Oct 31 13:34:37.051709 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:37.051674    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"
Oct 31 13:34:37.372822 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:34:37.372788    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34f0c7f_398d_4614_8851_913948829895.slice/crio-0fbe19690f2f007208609e2601135ab287cb7341481be0c87c5f97e623e409e5 WatchSource:0}: Error finding container 0fbe19690f2f007208609e2601135ab287cb7341481be0c87c5f97e623e409e5: Status 404 returned error can't find the container with id 0fbe19690f2f007208609e2601135ab287cb7341481be0c87c5f97e623e409e5
Oct 31 13:34:37.376432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:37.376398    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"]
Oct 31 13:34:37.441140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:37.441104    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:37.816613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:37.816575    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9" event={"ID":"e34f0c7f-398d-4614-8851-913948829895","Type":"ContainerStarted","Data":"1be9760b4fca4aa57c346eaae358b936f53cc5a718d7e515f41e2688cb861d1f"}
Oct 31 13:34:37.816838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:37.816638    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"
Oct 31 13:34:37.816838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:37.816662    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9" event={"ID":"e34f0c7f-398d-4614-8851-913948829895","Type":"ContainerStarted","Data":"0fbe19690f2f007208609e2601135ab287cb7341481be0c87c5f97e623e409e5"}
Oct 31 13:34:37.839957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:37.839894    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9" podStartSLOduration=2.8398550240000002 podStartE2EDuration="2.839855024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:37.838030123 +0000 UTC m=+1875.236378748" watchObservedRunningTime="2026-10-31 13:34:37.839855024 +0000 UTC m=+1875.238203649"
Oct 31 13:34:38.711892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:38.707745    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:39.035980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:39.035835    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:39.458950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:39.458588    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"
Oct 31 13:34:39.461236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:39.461147    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:39.635622 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:39.635566    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:40.941463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:40.941432    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:41.308908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:41.308831    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:42.483658 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:42.483628    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"]
Oct 31 13:34:42.488735 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:42.488717    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"
Oct 31 13:34:42.490260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:42.490235    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt5x9\" (UniqueName: \"kubernetes.io/projected/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-kube-api-access-qt5x9\") pod \"maas-api-revoke-sub-da34bff6-h7p22\" (UID: \"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"
Oct 31 13:34:42.490516 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:42.490321    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da34bff6-h7p22\" (UID: \"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"
Oct 31 13:34:42.490596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:42.490529    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da34bff6-h7p22\" (UID: \"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"
Oct 31 13:34:42.497301 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:42.497278    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:34:42.497594 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:42.497568    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:34:42.501365 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:42.501343    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"]
Oct 31 13:34:42.592011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:42.591978    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt5x9\" (UniqueName: \"kubernetes.io/projected/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-kube-api-access-qt5x9\") pod \"maas-api-revoke-sub-da34bff6-h7p22\" (UID: \"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"
Oct 31 13:34:42.592173 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:42.592067    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da34bff6-h7p22\" (UID: \"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"
Oct 31 13:34:42.592256 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:42.592206    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da34bff6-h7p22\" (UID: \"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"
Oct 31 13:34:42.592709 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:42.592686    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da34bff6-h7p22\" (UID: \"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"
Oct 31 13:34:42.594703 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:42.594677    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da34bff6-h7p22\" (UID: \"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"
Oct 31 13:34:42.605015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:42.604988    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt5x9\" (UniqueName: \"kubernetes.io/projected/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-kube-api-access-qt5x9\") pod \"maas-api-revoke-sub-da34bff6-h7p22\" (UID: \"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"
Oct 31 13:34:42.811403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:42.811309    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"
Oct 31 13:34:42.958012 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:42.957925    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:43.300686 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:34:43.300650    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fd0053_54bb_4fb3_85c5_50ef5c94e2ff.slice/crio-b6e8205d4af063c6f36dc44053a3f283dd4ef472418a3c3f9b0c0246b1742515 WatchSource:0}: Error finding container b6e8205d4af063c6f36dc44053a3f283dd4ef472418a3c3f9b0c0246b1742515: Status 404 returned error can't find the container with id b6e8205d4af063c6f36dc44053a3f283dd4ef472418a3c3f9b0c0246b1742515
Oct 31 13:34:43.301734 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:43.301698    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"]
Oct 31 13:34:43.355832 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:43.355807    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22" event={"ID":"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff","Type":"ContainerStarted","Data":"b6e8205d4af063c6f36dc44053a3f283dd4ef472418a3c3f9b0c0246b1742515"}
Oct 31 13:34:44.017393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:44.017354    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:44.034371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:44.034332    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"]
Oct 31 13:34:44.039062 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:44.039039    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768a7056-zg5j4"]
Oct 31 13:34:44.366541 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:44.366450    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22" event={"ID":"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff","Type":"ContainerStarted","Data":"f7f0ba019e1e63f5fc74f651df5ffcac74fa2ffabfd4502be23eacb5b775253e"}
Oct 31 13:34:44.387039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:44.386997    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22" podStartSLOduration=2.386984243 podStartE2EDuration="2.386984243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:44.385542127 +0000 UTC m=+1881.783890751" watchObservedRunningTime="2026-10-31 13:34:44.386984243 +0000 UTC m=+1881.785332866"
Oct 31 13:34:44.772783 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:44.772743    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26dd467a-54e2-4d31-a82e-74fcf4f39120" path="/var/lib/kubelet/pods/26dd467a-54e2-4d31-a82e-74fcf4f39120/volumes"
Oct 31 13:34:45.099942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:45.089570    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:45.377575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:45.377492    4428 generic.go:356] "Generic (PLEG): container finished" podID="08fd0053-54bb-4fb3-85c5-50ef5c94e2ff" containerID="f7f0ba019e1e63f5fc74f651df5ffcac74fa2ffabfd4502be23eacb5b775253e" exitCode=0
Oct 31 13:34:45.377735 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:45.377586    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22" event={"ID":"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff","Type":"ContainerDied","Data":"f7f0ba019e1e63f5fc74f651df5ffcac74fa2ffabfd4502be23eacb5b775253e"}
Oct 31 13:34:46.161748 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:46.161711    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:46.557723 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:46.557697    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"
Oct 31 13:34:46.689330 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:46.689302    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-maas-api-xks-ca\") pod \"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff\" (UID: \"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff\") "
Oct 31 13:34:46.689569 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:46.689553    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-kube-api-access-qt5x9\" (UniqueName: \"kubernetes.io/projected/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-kube-api-access-qt5x9\") pod \"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff\" (UID: \"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff\") "
Oct 31 13:34:46.689628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:46.689590    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-maas-api-openshift-ca\") pod \"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff\" (UID: \"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff\") "
Oct 31 13:34:46.690028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:46.690002    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-maas-api-openshift-ca" pod "08fd0053-54bb-4fb3-85c5-50ef5c94e2ff" (UID: "08fd0053-54bb-4fb3-85c5-50ef5c94e2ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:34:46.690402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:46.690382    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:34:46.691515 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:46.691493    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-maas-api-xks-ca" pod "08fd0053-54bb-4fb3-85c5-50ef5c94e2ff" (UID: "08fd0053-54bb-4fb3-85c5-50ef5c94e2ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:34:46.691789 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:46.691767    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-kube-api-access-qt5x9" pod "08fd0053-54bb-4fb3-85c5-50ef5c94e2ff" (UID: "08fd0053-54bb-4fb3-85c5-50ef5c94e2ff"). InnerVolumeSpecName "kube-api-access-qt5x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:34:46.792666 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:46.792638    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:34:46.792666 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:46.792668    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt5x9\" (UniqueName: \"kubernetes.io/projected/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff-kube-api-access-qt5x9\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:34:47.127672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:47.127628    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:47.396584 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:47.396500    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22" event={"ID":"08fd0053-54bb-4fb3-85c5-50ef5c94e2ff","Type":"ContainerDied","Data":"b6e8205d4af063c6f36dc44053a3f283dd4ef472418a3c3f9b0c0246b1742515"}
Oct 31 13:34:47.396997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:47.396595    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6e8205d4af063c6f36dc44053a3f283dd4ef472418a3c3f9b0c0246b1742515"
Oct 31 13:34:47.396997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:47.396514    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"
Oct 31 13:34:48.020901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:48.019537    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:48.600704 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:48.600676    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:48.744573 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:34:48.744537    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:34:48.876993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:48.859440    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:48.919121 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:48.919086    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"
Oct 31 13:34:49.032702 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:49.032667    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:49.213975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:49.211381    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:49.387908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:49.379283    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:50.037389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.037359    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"]
Oct 31 13:34:50.042087 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.042065    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4dc5e8b-hdfvx"]
Oct 31 13:34:50.210955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.210924    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"]
Oct 31 13:34:50.220893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.212440    4428 cpu_manager.go:414] "containerMap: removing container" podUID="08fd0053-54bb-4fb3-85c5-50ef5c94e2ff" containerName="revoke-keys"
Oct 31 13:34:50.221201 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.221176    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08fd0053-54bb-4fb3-85c5-50ef5c94e2ff" containerName="revoke-keys"
Oct 31 13:34:50.221668 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.221644    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="08fd0053-54bb-4fb3-85c5-50ef5c94e2ff" containerName="revoke-keys"
Oct 31 13:34:50.231891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.215975    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:50.231891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.230817    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"
Oct 31 13:34:50.238101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.238070    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:34:50.238613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.238591    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:34:50.272116 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.272009    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"]
Oct 31 13:34:50.339906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.337152    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/455a838e-c5ee-460e-80db-c8b3bf879c28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-466002-6v6zq\" (UID: \"455a838e-c5ee-460e-80db-c8b3bf879c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"
Oct 31 13:34:50.339906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.337337    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/455a838e-c5ee-460e-80db-c8b3bf879c28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-466002-6v6zq\" (UID: \"455a838e-c5ee-460e-80db-c8b3bf879c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"
Oct 31 13:34:50.339906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.337480    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59fl9\" (UniqueName: \"kubernetes.io/projected/455a838e-c5ee-460e-80db-c8b3bf879c28-kube-api-access-59fl9\") pod \"maas-api-revoke-keys-e2e-api-a-466002-6v6zq\" (UID: \"455a838e-c5ee-460e-80db-c8b3bf879c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"
Oct 31 13:34:50.438788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.438753    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/455a838e-c5ee-460e-80db-c8b3bf879c28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-466002-6v6zq\" (UID: \"455a838e-c5ee-460e-80db-c8b3bf879c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"
Oct 31 13:34:50.438995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.438850    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59fl9\" (UniqueName: \"kubernetes.io/projected/455a838e-c5ee-460e-80db-c8b3bf879c28-kube-api-access-59fl9\") pod \"maas-api-revoke-keys-e2e-api-a-466002-6v6zq\" (UID: \"455a838e-c5ee-460e-80db-c8b3bf879c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"
Oct 31 13:34:50.439075 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.439059    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/455a838e-c5ee-460e-80db-c8b3bf879c28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-466002-6v6zq\" (UID: \"455a838e-c5ee-460e-80db-c8b3bf879c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"
Oct 31 13:34:50.439544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.439521    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/455a838e-c5ee-460e-80db-c8b3bf879c28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-466002-6v6zq\" (UID: \"455a838e-c5ee-460e-80db-c8b3bf879c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"
Oct 31 13:34:50.441639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.441612    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/455a838e-c5ee-460e-80db-c8b3bf879c28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-466002-6v6zq\" (UID: \"455a838e-c5ee-460e-80db-c8b3bf879c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"
Oct 31 13:34:50.460608 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.460580    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59fl9\" (UniqueName: \"kubernetes.io/projected/455a838e-c5ee-460e-80db-c8b3bf879c28-kube-api-access-59fl9\") pod \"maas-api-revoke-keys-e2e-api-a-466002-6v6zq\" (UID: \"455a838e-c5ee-460e-80db-c8b3bf879c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"
Oct 31 13:34:50.654940 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.652161    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"
Oct 31 13:34:50.800297 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:50.800253    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="221eeffa-0e13-4f2b-b071-954dc973142a" path="/var/lib/kubelet/pods/221eeffa-0e13-4f2b-b071-954dc973142a/volumes"
Oct 31 13:34:51.028616 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:51.028582    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"]
Oct 31 13:34:51.032841 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:51.032809    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-967a9954-9tldm"]
Oct 31 13:34:51.092193 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:34:51.092161    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455a838e_c5ee_460e_80db_c8b3bf879c28.slice/crio-7e30097d2e684d4bf38f9aff3f3897b359b58be3a10ffa0659ee57b27f024425 WatchSource:0}: Error finding container 7e30097d2e684d4bf38f9aff3f3897b359b58be3a10ffa0659ee57b27f024425: Status 404 returned error can't find the container with id 7e30097d2e684d4bf38f9aff3f3897b359b58be3a10ffa0659ee57b27f024425
Oct 31 13:34:51.092723 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:51.092699    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"]
Oct 31 13:34:51.529037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:51.528998    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq" event={"ID":"455a838e-c5ee-460e-80db-c8b3bf879c28","Type":"ContainerStarted","Data":"984ea124b5d39fcd2e0dbb76bf71260ae041f1b7ea3e399d86d8935b98881cae"}
Oct 31 13:34:51.529223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:51.529101    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq" event={"ID":"455a838e-c5ee-460e-80db-c8b3bf879c28","Type":"ContainerStarted","Data":"7e30097d2e684d4bf38f9aff3f3897b359b58be3a10ffa0659ee57b27f024425"}
Oct 31 13:34:51.554189 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:51.554143    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq" podStartSLOduration=1.5541307720000002 podStartE2EDuration="1.554130772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:51.551063294 +0000 UTC m=+1888.949411919" watchObservedRunningTime="2026-10-31 13:34:51.554130772 +0000 UTC m=+1888.952479396"
Oct 31 13:34:51.681119 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:51.681030    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:52.548468 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:52.548425    4428 generic.go:356] "Generic (PLEG): container finished" podID="455a838e-c5ee-460e-80db-c8b3bf879c28" containerID="984ea124b5d39fcd2e0dbb76bf71260ae041f1b7ea3e399d86d8935b98881cae" exitCode=0
Oct 31 13:34:52.548943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:52.548581    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq" event={"ID":"455a838e-c5ee-460e-80db-c8b3bf879c28","Type":"ContainerDied","Data":"984ea124b5d39fcd2e0dbb76bf71260ae041f1b7ea3e399d86d8935b98881cae"}
Oct 31 13:34:52.739893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:52.735710    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:52.810946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:52.802350    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd3967ed-136c-41fe-9f49-e865b6887dd7" path="/var/lib/kubelet/pods/cd3967ed-136c-41fe-9f49-e865b6887dd7/volumes"
Oct 31 13:34:52.917771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:52.917727    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:53.096790 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:53.096713    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:53.892066 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:53.892036    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"
Oct 31 13:34:54.044799 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.044770    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/455a838e-c5ee-460e-80db-c8b3bf879c28-kube-api-access-59fl9\" (UniqueName: \"kubernetes.io/projected/455a838e-c5ee-460e-80db-c8b3bf879c28-kube-api-access-59fl9\") pod \"455a838e-c5ee-460e-80db-c8b3bf879c28\" (UID: \"455a838e-c5ee-460e-80db-c8b3bf879c28\") "
Oct 31 13:34:54.045061 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.045039    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/455a838e-c5ee-460e-80db-c8b3bf879c28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/455a838e-c5ee-460e-80db-c8b3bf879c28-maas-api-xks-ca\") pod \"455a838e-c5ee-460e-80db-c8b3bf879c28\" (UID: \"455a838e-c5ee-460e-80db-c8b3bf879c28\") "
Oct 31 13:34:54.045272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.045255    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/455a838e-c5ee-460e-80db-c8b3bf879c28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/455a838e-c5ee-460e-80db-c8b3bf879c28-maas-api-openshift-ca\") pod \"455a838e-c5ee-460e-80db-c8b3bf879c28\" (UID: \"455a838e-c5ee-460e-80db-c8b3bf879c28\") "
Oct 31 13:34:54.045686 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.045655    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/455a838e-c5ee-460e-80db-c8b3bf879c28-maas-api-openshift-ca" pod "455a838e-c5ee-460e-80db-c8b3bf879c28" (UID: "455a838e-c5ee-460e-80db-c8b3bf879c28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:34:54.046102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.046081    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/455a838e-c5ee-460e-80db-c8b3bf879c28-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:34:54.047134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.047108    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/455a838e-c5ee-460e-80db-c8b3bf879c28-maas-api-xks-ca" pod "455a838e-c5ee-460e-80db-c8b3bf879c28" (UID: "455a838e-c5ee-460e-80db-c8b3bf879c28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:34:54.047274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.047252    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455a838e-c5ee-460e-80db-c8b3bf879c28-kube-api-access-59fl9" pod "455a838e-c5ee-460e-80db-c8b3bf879c28" (UID: "455a838e-c5ee-460e-80db-c8b3bf879c28"). InnerVolumeSpecName "kube-api-access-59fl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:34:54.181500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.180464    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59fl9\" (UniqueName: \"kubernetes.io/projected/455a838e-c5ee-460e-80db-c8b3bf879c28-kube-api-access-59fl9\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:34:54.181500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.180508    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/455a838e-c5ee-460e-80db-c8b3bf879c28-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:34:54.193980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.182485    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:54.344368 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.344257    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:54.414782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.414736    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"]
Oct 31 13:34:54.416539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.416509    4428 cpu_manager.go:414] "containerMap: removing container" podUID="455a838e-c5ee-460e-80db-c8b3bf879c28" containerName="revoke-keys"
Oct 31 13:34:54.416539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.416539    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="455a838e-c5ee-460e-80db-c8b3bf879c28" containerName="revoke-keys"
Oct 31 13:34:54.417014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.416992    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="455a838e-c5ee-460e-80db-c8b3bf879c28" containerName="revoke-keys"
Oct 31 13:34:54.425345 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.424460    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"
Oct 31 13:34:54.436843 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.436808    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-580f24\""
Oct 31 13:34:54.455900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.441672    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"]
Oct 31 13:34:54.530936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.471676    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-580f24-dockercfg-pb5zd\""
Oct 31 13:34:54.624975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.617914    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndq7c\" (UniqueName: \"kubernetes.io/projected/767378d1-3634-4db5-873f-7cc9fc3fb5a9-kube-api-access-ndq7c\") pod \"payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9\" (UID: \"767378d1-3634-4db5-873f-7cc9fc3fb5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"
Oct 31 13:34:54.624975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.618241    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767378d1-3634-4db5-873f-7cc9fc3fb5a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9\" (UID: \"767378d1-3634-4db5-873f-7cc9fc3fb5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"
Oct 31 13:34:54.726817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.726528    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767378d1-3634-4db5-873f-7cc9fc3fb5a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9\" (UID: \"767378d1-3634-4db5-873f-7cc9fc3fb5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"
Oct 31 13:34:54.726817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.726764    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndq7c\" (UniqueName: \"kubernetes.io/projected/767378d1-3634-4db5-873f-7cc9fc3fb5a9-kube-api-access-ndq7c\") pod \"payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9\" (UID: \"767378d1-3634-4db5-873f-7cc9fc3fb5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"
Oct 31 13:34:54.753267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.752747    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767378d1-3634-4db5-873f-7cc9fc3fb5a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9\" (UID: \"767378d1-3634-4db5-873f-7cc9fc3fb5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"
Oct 31 13:34:54.871907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:54.857301    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndq7c\" (UniqueName: \"kubernetes.io/projected/767378d1-3634-4db5-873f-7cc9fc3fb5a9-kube-api-access-ndq7c\") pod \"payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9\" (UID: \"767378d1-3634-4db5-873f-7cc9fc3fb5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"
Oct 31 13:34:55.021318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:55.021288    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"
Oct 31 13:34:55.022201 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:55.021854    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq" event={"ID":"455a838e-c5ee-460e-80db-c8b3bf879c28","Type":"ContainerDied","Data":"7e30097d2e684d4bf38f9aff3f3897b359b58be3a10ffa0659ee57b27f024425"}
Oct 31 13:34:55.022201 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:55.021931    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e30097d2e684d4bf38f9aff3f3897b359b58be3a10ffa0659ee57b27f024425"
Oct 31 13:34:55.022201 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:55.022015    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"
Oct 31 13:34:55.353721 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:55.353623    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:55.386894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:55.383565    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"]
Oct 31 13:34:56.032309 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:56.032269    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9" event={"ID":"767378d1-3634-4db5-873f-7cc9fc3fb5a9","Type":"ContainerStarted","Data":"6c5891d5958600fec530f7ae5e814a03f3e3f88aa36dd3f143a3a31ebfd14723"}
Oct 31 13:34:56.032695 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:56.032332    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9" event={"ID":"767378d1-3634-4db5-873f-7cc9fc3fb5a9","Type":"ContainerStarted","Data":"ff67fc16dc9ce75c65f4758e7d38868198a894160aba50ac404b2d280fe5d4aa"}
Oct 31 13:34:56.032695 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:56.032486    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"
Oct 31 13:34:56.056191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:56.056148    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9" podStartSLOduration=2.056135453 podStartE2EDuration="2.056135453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:56.051314159 +0000 UTC m=+1893.449662786" watchObservedRunningTime="2026-10-31 13:34:56.056135453 +0000 UTC m=+1893.454484132"
Oct 31 13:34:56.111535 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:56.111512    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:57.096310 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:57.093813    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:58.009893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:58.009071    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:58.091186 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:58.090305    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"]
Oct 31 13:34:58.136260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:58.135256    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cbf5582-v49ql"]
Oct 31 13:34:58.894897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:58.894736    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:34:58.913890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:58.895187    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2160f0a2-8ac4-46cf-841b-addfbb0e2cf8" path="/var/lib/kubelet/pods/2160f0a2-8ac4-46cf-841b-addfbb0e2cf8/volumes"
Oct 31 13:34:59.771157 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:34:59.771126    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:00.330263 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:00.327908    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:00.440566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:00.440529    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"]
Oct 31 13:35:00.447766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:00.447737    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-pwhhs"]
Oct 31 13:35:00.586408 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:00.584599    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:00.658838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:00.658810    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"]
Oct 31 13:35:00.761546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:00.761510    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba5687b-c871-4a60-9c57-42170ccb5c7d" path="/var/lib/kubelet/pods/fba5687b-c871-4a60-9c57-42170ccb5c7d/volumes"
Oct 31 13:35:00.791386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:00.791356    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:00.954940 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:00.954272    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:01.697680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:01.697340    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:02.621206 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:02.621132    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:03.409899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:03.408527    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:04.197734 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:04.197701    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:04.374146 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:04.374104    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:04.721771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:04.721709    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:05.050549 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:05.050390    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:05.659243 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:05.659187    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j" podUID="d77bf054-b344-4ea7-b809-ea8ea990aef6" containerName="payload-pre-processing" containerID="cri-o://d08fb901c5c3403583cf1587c95579da9b62d1361cb0991420fa07f614d97bd9" gracePeriod=25
Oct 31 13:35:05.835898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:05.834065    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:06.124926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.124860    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"
Oct 31 13:35:06.227211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.227176    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77bf054-b344-4ea7-b809-ea8ea990aef6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77bf054-b344-4ea7-b809-ea8ea990aef6-plugins-config-volume\") pod \"d77bf054-b344-4ea7-b809-ea8ea990aef6\" (UID: \"d77bf054-b344-4ea7-b809-ea8ea990aef6\") "
Oct 31 13:35:06.227511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.227495    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77bf054-b344-4ea7-b809-ea8ea990aef6-kube-api-access-qzvj9\" (UniqueName: \"kubernetes.io/projected/d77bf054-b344-4ea7-b809-ea8ea990aef6-kube-api-access-qzvj9\") pod \"d77bf054-b344-4ea7-b809-ea8ea990aef6\" (UID: \"d77bf054-b344-4ea7-b809-ea8ea990aef6\") "
Oct 31 13:35:06.227602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.227580    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77bf054-b344-4ea7-b809-ea8ea990aef6-plugins-config-volume" pod "d77bf054-b344-4ea7-b809-ea8ea990aef6" (UID: "d77bf054-b344-4ea7-b809-ea8ea990aef6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:35:06.228017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.227997    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77bf054-b344-4ea7-b809-ea8ea990aef6-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:06.229755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.229730    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77bf054-b344-4ea7-b809-ea8ea990aef6-kube-api-access-qzvj9" pod "d77bf054-b344-4ea7-b809-ea8ea990aef6" (UID: "d77bf054-b344-4ea7-b809-ea8ea990aef6"). InnerVolumeSpecName "kube-api-access-qzvj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:35:06.330085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.330009    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzvj9\" (UniqueName: \"kubernetes.io/projected/d77bf054-b344-4ea7-b809-ea8ea990aef6-kube-api-access-qzvj9\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:06.559918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.559855    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:06.612155 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.612058    4428 generic.go:356] "Generic (PLEG): container finished" podID="d77bf054-b344-4ea7-b809-ea8ea990aef6" containerID="d08fb901c5c3403583cf1587c95579da9b62d1361cb0991420fa07f614d97bd9" exitCode=0
Oct 31 13:35:06.612648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.612614    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j" event={"ID":"d77bf054-b344-4ea7-b809-ea8ea990aef6","Type":"ContainerDied","Data":"d08fb901c5c3403583cf1587c95579da9b62d1361cb0991420fa07f614d97bd9"}
Oct 31 13:35:06.621348 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.615994    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"
Oct 31 13:35:06.621670 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.621639    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j" event={"ID":"d77bf054-b344-4ea7-b809-ea8ea990aef6","Type":"ContainerDied","Data":"f08efe23c28a68c840e8ac743a58c1ac845af420329d9b053c05a017ba83fbfe"}
Oct 31 13:35:06.621814 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.621802    4428 scope.go:122] "RemoveContainer" containerID="d08fb901c5c3403583cf1587c95579da9b62d1361cb0991420fa07f614d97bd9"
Oct 31 13:35:06.674921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.674778    4428 scope.go:122] "RemoveContainer" containerID="d08fb901c5c3403583cf1587c95579da9b62d1361cb0991420fa07f614d97bd9"
Oct 31 13:35:06.675183 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:35:06.675159    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d08fb901c5c3403583cf1587c95579da9b62d1361cb0991420fa07f614d97bd9\": container with ID starting with d08fb901c5c3403583cf1587c95579da9b62d1361cb0991420fa07f614d97bd9 not found: ID does not exist" containerID="d08fb901c5c3403583cf1587c95579da9b62d1361cb0991420fa07f614d97bd9"
Oct 31 13:35:06.675225 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.675194    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08fb901c5c3403583cf1587c95579da9b62d1361cb0991420fa07f614d97bd9"} err="failed to get container status \"d08fb901c5c3403583cf1587c95579da9b62d1361cb0991420fa07f614d97bd9\": rpc error: code = NotFound desc = could not find container \"d08fb901c5c3403583cf1587c95579da9b62d1361cb0991420fa07f614d97bd9\": container with ID starting with d08fb901c5c3403583cf1587c95579da9b62d1361cb0991420fa07f614d97bd9 not found: ID does not exist"
Oct 31 13:35:06.692153 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.692128    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"]
Oct 31 13:35:06.696859 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.696836    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-466002-7b88dbfff9-sph4j"]
Oct 31 13:35:06.757400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:06.757369    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d77bf054-b344-4ea7-b809-ea8ea990aef6" path="/var/lib/kubelet/pods/d77bf054-b344-4ea7-b809-ea8ea990aef6/volumes"
Oct 31 13:35:07.101629 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:07.101595    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"
Oct 31 13:35:07.603016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:07.602963    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:07.761429 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:07.761395    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:07.936402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:07.936323    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:08.230952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:08.230916    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"]
Oct 31 13:35:08.231691 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:08.231677    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d77bf054-b344-4ea7-b809-ea8ea990aef6" containerName="payload-pre-processing"
Oct 31 13:35:08.231737 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:08.231693    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d77bf054-b344-4ea7-b809-ea8ea990aef6" containerName="payload-pre-processing"
Oct 31 13:35:08.231834 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:08.231826    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d77bf054-b344-4ea7-b809-ea8ea990aef6" containerName="payload-pre-processing"
Oct 31 13:35:08.237527 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:08.237503    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"
Oct 31 13:35:08.240710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:08.240687    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:08.245158 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:08.245123    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-79f4fb0f\""
Oct 31 13:35:08.245279 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:08.245175    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-79f4fb0f-dockercfg-zc24t\""
Oct 31 13:35:08.253025 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:08.253000    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"]
Oct 31 13:35:08.583811 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:08.583712    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sxjw\" (UniqueName: \"kubernetes.io/projected/6cda6b31-2ad2-4248-aa7b-7ca7a25ef681-kube-api-access-7sxjw\") pod \"payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88\" (UID: \"6cda6b31-2ad2-4248-aa7b-7ca7a25ef681\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"
Oct 31 13:35:08.643926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:08.616979    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:08.653706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:08.652382    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cda6b31-2ad2-4248-aa7b-7ca7a25ef681-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88\" (UID: \"6cda6b31-2ad2-4248-aa7b-7ca7a25ef681\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"
Oct 31 13:35:08.784893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:08.781536    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cda6b31-2ad2-4248-aa7b-7ca7a25ef681-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88\" (UID: \"6cda6b31-2ad2-4248-aa7b-7ca7a25ef681\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"
Oct 31 13:35:08.784893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:08.781764    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sxjw\" (UniqueName: \"kubernetes.io/projected/6cda6b31-2ad2-4248-aa7b-7ca7a25ef681-kube-api-access-7sxjw\") pod \"payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88\" (UID: \"6cda6b31-2ad2-4248-aa7b-7ca7a25ef681\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"
Oct 31 13:35:08.877982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:08.804846    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cda6b31-2ad2-4248-aa7b-7ca7a25ef681-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88\" (UID: \"6cda6b31-2ad2-4248-aa7b-7ca7a25ef681\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"
Oct 31 13:35:08.877982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:08.810973    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sxjw\" (UniqueName: \"kubernetes.io/projected/6cda6b31-2ad2-4248-aa7b-7ca7a25ef681-kube-api-access-7sxjw\") pod \"payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88\" (UID: \"6cda6b31-2ad2-4248-aa7b-7ca7a25ef681\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"
Oct 31 13:35:08.877982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:08.859467    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"
Oct 31 13:35:09.216532 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:09.216437    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:09.295891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:09.295836    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"]
Oct 31 13:35:09.858242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:09.858206    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88" event={"ID":"6cda6b31-2ad2-4248-aa7b-7ca7a25ef681","Type":"ContainerStarted","Data":"3bac2c5ff9fdff976cff5d8a5fcfacbc213980682c8f05f82d9389cd9a1b22ba"}
Oct 31 13:35:09.858664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:09.858269    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88" event={"ID":"6cda6b31-2ad2-4248-aa7b-7ca7a25ef681","Type":"ContainerStarted","Data":"d0af413a64add90ce8ea29ab2474e3edd03d15db8182e2bd932afa8287c1785b"}
Oct 31 13:35:09.858664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:09.858364    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"
Oct 31 13:35:09.881627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:09.881575    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88" podStartSLOduration=1.881560608 podStartE2EDuration="1.881560608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:09.878150157 +0000 UTC m=+1907.276498781" watchObservedRunningTime="2026-10-31 13:35:09.881560608 +0000 UTC m=+1907.279909232"
Oct 31 13:35:10.090459 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:10.089820    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:11.120669 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:11.112440    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:12.015036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:12.014994    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:12.796895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:12.794279    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:13.137931 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:13.137836    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:13.292402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:13.292330    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:13.542714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:13.542673    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:14.143962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:14.143911    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:14.338903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:14.338210    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:14.649687 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:14.649635    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"]
Oct 31 13:35:14.649980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:14.649692    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:14.649980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:14.649922    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl" podUID="a4853846-205d-4d62-a056-db336dee24cb" containerName="istio-proxy" containerID="cri-o://c87a1754d7427fa96dcfc16168934d30dfde33085f41744efb5f8460046f5e5d" gracePeriod=30
Oct 31 13:35:14.661466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:14.661440    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"]
Oct 31 13:35:14.661681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:14.661661    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" containerID="cri-o://05e998ce72c6c5ba640a86174ae695bf5a17360f8e8a63ad64a1951932ab5fa4" gracePeriod=30
Oct 31 13:35:14.825893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:14.821376    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:15.020893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:15.013382    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:15.645449 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:15.645417    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:15.816992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:15.816431    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:15.985403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:15.985372    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:16.308961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:16.305338    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"]
Oct 31 13:35:16.330894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:16.327475    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"
Oct 31 13:35:16.340889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:16.332819    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:35:16.340889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:16.336039    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"]
Oct 31 13:35:16.340889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:16.336258    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:35:16.462887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:16.456681    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-75c956-4vj88\" (UID: \"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"
Oct 31 13:35:16.462887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:16.457146    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-75c956-4vj88\" (UID: \"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"
Oct 31 13:35:16.462887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:16.457298    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqxld\" (UniqueName: \"kubernetes.io/projected/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-kube-api-access-xqxld\") pod \"maas-api-revoke-keys-e2e-api-a-75c956-4vj88\" (UID: \"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"
Oct 31 13:35:16.562393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:16.562290    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-75c956-4vj88\" (UID: \"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"
Oct 31 13:35:16.562711 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:16.562684    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqxld\" (UniqueName: \"kubernetes.io/projected/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-kube-api-access-xqxld\") pod \"maas-api-revoke-keys-e2e-api-a-75c956-4vj88\" (UID: \"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"
Oct 31 13:35:16.563069 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:16.563047    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-75c956-4vj88\" (UID: \"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"
Oct 31 13:35:16.564410 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:16.564366    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-75c956-4vj88\" (UID: \"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"
Oct 31 13:35:16.597159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:16.597126    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-75c956-4vj88\" (UID: \"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"
Oct 31 13:35:16.608968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:16.599590    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqxld\" (UniqueName: \"kubernetes.io/projected/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-kube-api-access-xqxld\") pod \"maas-api-revoke-keys-e2e-api-a-75c956-4vj88\" (UID: \"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"
Oct 31 13:35:16.772586 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:16.772550    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"
Oct 31 13:35:17.134988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:17.129764    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"]
Oct 31 13:35:17.156360 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:35:17.156325    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice/crio-91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15 WatchSource:0}: Error finding container 91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15: Status 404 returned error can't find the container with id 91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15
Oct 31 13:35:17.204078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:17.202573    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:18.040918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:18.040882    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"]
Oct 31 13:35:18.045534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:18.045505    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59aafb2f-dj64l"]
Oct 31 13:35:18.132170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:18.132137    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88" event={"ID":"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e","Type":"ContainerStarted","Data":"6cb1a2fc3a59ff0cfafdbdcb2b4435a01b4e1fb76170219bd82c3a0d63e0349e"}
Oct 31 13:35:18.132332 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:18.132191    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88" event={"ID":"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e","Type":"ContainerStarted","Data":"91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15"}
Oct 31 13:35:18.153570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:18.153521    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88" podStartSLOduration=2.153503411 podStartE2EDuration="2.153503411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:18.151523369 +0000 UTC m=+1915.549871988" watchObservedRunningTime="2026-10-31 13:35:18.153503411 +0000 UTC m=+1915.551852039"
Oct 31 13:35:18.233895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:18.232216    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:18.742350 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:35:18.742316    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:35:18.757229 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:18.757201    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648da99d-008c-4647-a4c5-0e40d1b932a8" path="/var/lib/kubelet/pods/648da99d-008c-4647-a4c5-0e40d1b932a8/volumes"
Oct 31 13:35:18.787538 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:18.787503    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:35:18.787661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:18.787561    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:35:19.142420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:19.142331    4428 generic.go:356] "Generic (PLEG): container finished" podID="080d9f1e-4bc4-4413-9cf9-80e1b7ac941e" containerID="6cb1a2fc3a59ff0cfafdbdcb2b4435a01b4e1fb76170219bd82c3a0d63e0349e" exitCode=0
Oct 31 13:35:19.142420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:19.142388    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88" event={"ID":"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e","Type":"ContainerDied","Data":"6cb1a2fc3a59ff0cfafdbdcb2b4435a01b4e1fb76170219bd82c3a0d63e0349e"}
Oct 31 13:35:19.282574 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:19.282513    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:19.489894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:19.463765    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:20.031777 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.031713    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:35:20.133966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.131181    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-credential-socket\") pod \"095761dd-4e84-4051-b472-b85d43d444c9\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") "
Oct 31 13:35:20.133966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.131293    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/095761dd-4e84-4051-b472-b85d43d444c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095761dd-4e84-4051-b472-b85d43d444c9-istio-podinfo\") pod \"095761dd-4e84-4051-b472-b85d43d444c9\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") "
Oct 31 13:35:20.133966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.131355    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-workload-certs\") pod \"095761dd-4e84-4051-b472-b85d43d444c9\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") "
Oct 31 13:35:20.133966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.131394    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-istio-envoy\") pod \"095761dd-4e84-4051-b472-b85d43d444c9\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") "
Oct 31 13:35:20.133966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.131469    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-workload-socket\") pod \"095761dd-4e84-4051-b472-b85d43d444c9\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") "
Oct 31 13:35:20.133966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.131573    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/095761dd-4e84-4051-b472-b85d43d444c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095761dd-4e84-4051-b472-b85d43d444c9-istiod-ca-cert\") pod \"095761dd-4e84-4051-b472-b85d43d444c9\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") "
Oct 31 13:35:20.133966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.131644    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095761dd-4e84-4051-b472-b85d43d444c9-kube-api-access-9hh67\" (UniqueName: \"kubernetes.io/projected/095761dd-4e84-4051-b472-b85d43d444c9-kube-api-access-9hh67\") pod \"095761dd-4e84-4051-b472-b85d43d444c9\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") "
Oct 31 13:35:20.133966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.131708    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-istio-data\") pod \"095761dd-4e84-4051-b472-b85d43d444c9\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") "
Oct 31 13:35:20.133966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.131929    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095761dd-4e84-4051-b472-b85d43d444c9-istio-token\" (UniqueName: \"kubernetes.io/projected/095761dd-4e84-4051-b472-b85d43d444c9-istio-token\") pod \"095761dd-4e84-4051-b472-b85d43d444c9\" (UID: \"095761dd-4e84-4051-b472-b85d43d444c9\") "
Oct 31 13:35:20.159555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.159247    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/095761dd-4e84-4051-b472-b85d43d444c9-istiod-ca-cert" pod "095761dd-4e84-4051-b472-b85d43d444c9" (UID: "095761dd-4e84-4051-b472-b85d43d444c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:35:20.183101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.160101    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-credential-socket" pod "095761dd-4e84-4051-b472-b85d43d444c9" (UID: "095761dd-4e84-4051-b472-b85d43d444c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:35:20.183101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.161335    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-workload-certs" pod "095761dd-4e84-4051-b472-b85d43d444c9" (UID: "095761dd-4e84-4051-b472-b85d43d444c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:35:20.183101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.161594    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-workload-socket" pod "095761dd-4e84-4051-b472-b85d43d444c9" (UID: "095761dd-4e84-4051-b472-b85d43d444c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:35:20.183101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.161707    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095761dd-4e84-4051-b472-b85d43d444c9-istio-token" pod "095761dd-4e84-4051-b472-b85d43d444c9" (UID: "095761dd-4e84-4051-b472-b85d43d444c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:35:20.183101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.161912    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-istio-data" pod "095761dd-4e84-4051-b472-b85d43d444c9" (UID: "095761dd-4e84-4051-b472-b85d43d444c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:35:20.183101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.183023    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/095761dd-4e84-4051-b472-b85d43d444c9-istio-podinfo" pod "095761dd-4e84-4051-b472-b85d43d444c9" (UID: "095761dd-4e84-4051-b472-b85d43d444c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:35:20.192063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.185547    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095761dd-4e84-4051-b472-b85d43d444c9-kube-api-access-9hh67" pod "095761dd-4e84-4051-b472-b85d43d444c9" (UID: "095761dd-4e84-4051-b472-b85d43d444c9"). InnerVolumeSpecName "kube-api-access-9hh67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:35:20.244920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.236554    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:20.244920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.236597    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095761dd-4e84-4051-b472-b85d43d444c9-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:20.244920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.236613    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hh67\" (UniqueName: \"kubernetes.io/projected/095761dd-4e84-4051-b472-b85d43d444c9-kube-api-access-9hh67\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:20.244920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.236630    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:20.244920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.236644    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095761dd-4e84-4051-b472-b85d43d444c9-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:20.244920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.236656    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:20.244920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.236669    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095761dd-4e84-4051-b472-b85d43d444c9-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:20.244920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.236683    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:20.251090 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.250980    4428 generic.go:356] "Generic (PLEG): container finished" podID="a4853846-205d-4d62-a056-db336dee24cb" containerID="c87a1754d7427fa96dcfc16168934d30dfde33085f41744efb5f8460046f5e5d" exitCode=0
Oct 31 13:35:20.251597 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.251533    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl" event={"ID":"a4853846-205d-4d62-a056-db336dee24cb","Type":"ContainerDied","Data":"c87a1754d7427fa96dcfc16168934d30dfde33085f41744efb5f8460046f5e5d"}
Oct 31 13:35:20.292064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.276223    4428 generic.go:356] "Generic (PLEG): container finished" podID="095761dd-4e84-4051-b472-b85d43d444c9" containerID="05e998ce72c6c5ba640a86174ae695bf5a17360f8e8a63ad64a1951932ab5fa4" exitCode=0
Oct 31 13:35:20.292064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.276923    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"
Oct 31 13:35:20.292064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.277521    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" event={"ID":"095761dd-4e84-4051-b472-b85d43d444c9","Type":"ContainerDied","Data":"05e998ce72c6c5ba640a86174ae695bf5a17360f8e8a63ad64a1951932ab5fa4"}
Oct 31 13:35:20.305654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.305578    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-istio-envoy" pod "095761dd-4e84-4051-b472-b85d43d444c9" (UID: "095761dd-4e84-4051-b472-b85d43d444c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:35:20.306092 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.305912    4428 scope.go:122] "RemoveContainer" containerID="05e998ce72c6c5ba640a86174ae695bf5a17360f8e8a63ad64a1951932ab5fa4"
Oct 31 13:35:20.311956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.306922    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww" event={"ID":"095761dd-4e84-4051-b472-b85d43d444c9","Type":"ContainerDied","Data":"afc2d23b1e6a9c7f115f8514138f3b9047a638e27b71ccd7560fd04c5f8e00a5"}
Oct 31 13:35:20.341896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.339743    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095761dd-4e84-4051-b472-b85d43d444c9-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:20.504592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.492716    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:20.587473 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.555059    4428 scope.go:122] "RemoveContainer" containerID="05e998ce72c6c5ba640a86174ae695bf5a17360f8e8a63ad64a1951932ab5fa4"
Oct 31 13:35:20.590196 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:35:20.587707    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e998ce72c6c5ba640a86174ae695bf5a17360f8e8a63ad64a1951932ab5fa4\": container with ID starting with 05e998ce72c6c5ba640a86174ae695bf5a17360f8e8a63ad64a1951932ab5fa4 not found: ID does not exist" containerID="05e998ce72c6c5ba640a86174ae695bf5a17360f8e8a63ad64a1951932ab5fa4"
Oct 31 13:35:20.590196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.587752    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e998ce72c6c5ba640a86174ae695bf5a17360f8e8a63ad64a1951932ab5fa4"} err="failed to get container status \"05e998ce72c6c5ba640a86174ae695bf5a17360f8e8a63ad64a1951932ab5fa4\": rpc error: code = NotFound desc = could not find container \"05e998ce72c6c5ba640a86174ae695bf5a17360f8e8a63ad64a1951932ab5fa4\": container with ID starting with 05e998ce72c6c5ba640a86174ae695bf5a17360f8e8a63ad64a1951932ab5fa4 not found: ID does not exist"
Oct 31 13:35:20.840166 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.840083    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"]
Oct 31 13:35:20.848561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.848531    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-tdnww"]
Oct 31 13:35:20.881189 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.881162    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:20.910726 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.910696    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"]
Oct 31 13:35:20.911628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.911606    4428 cpu_manager.go:414] "containerMap: removing container" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy"
Oct 31 13:35:20.911699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.911630    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy"
Oct 31 13:35:20.911764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.911753    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="095761dd-4e84-4051-b472-b85d43d444c9" containerName="istio-proxy"
Oct 31 13:35:20.918124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.918095    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"
Oct 31 13:35:20.929766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.928633    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6aa924\""
Oct 31 13:35:20.929766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.928852    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6aa924-dockercfg-dx5fj\""
Oct 31 13:35:20.931216 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.931140    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"]
Oct 31 13:35:20.958928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.950601    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"]
Oct 31 13:35:20.958928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.958444    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"]
Oct 31 13:35:20.958928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.958554    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"
Oct 31 13:35:20.962646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:20.959801    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"
Oct 31 13:35:21.012561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.012529    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r84bq\" (UniqueName: \"kubernetes.io/projected/7b51cfae-67bd-4df7-a6da-802ec754ffe3-kube-api-access-r84bq\") pod \"payload-processing-e2e-api-a-6aa924-775b66d469-82hlk\" (UID: \"7b51cfae-67bd-4df7-a6da-802ec754ffe3\") " pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"
Oct 31 13:35:21.012826 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.012797    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51ddd17-4a94-468b-9ee7-c2f0de9b4528-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p\" (UID: \"b51ddd17-4a94-468b-9ee7-c2f0de9b4528\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"
Oct 31 13:35:21.012981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.012956    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b51cfae-67bd-4df7-a6da-802ec754ffe3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6aa924-775b66d469-82hlk\" (UID: \"7b51cfae-67bd-4df7-a6da-802ec754ffe3\") " pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"
Oct 31 13:35:21.013152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.013135    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24k24\" (UniqueName: \"kubernetes.io/projected/b51ddd17-4a94-468b-9ee7-c2f0de9b4528-kube-api-access-24k24\") pod \"payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p\" (UID: \"b51ddd17-4a94-468b-9ee7-c2f0de9b4528\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"
Oct 31 13:35:21.124321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.115217    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24k24\" (UniqueName: \"kubernetes.io/projected/b51ddd17-4a94-468b-9ee7-c2f0de9b4528-kube-api-access-24k24\") pod \"payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p\" (UID: \"b51ddd17-4a94-468b-9ee7-c2f0de9b4528\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"
Oct 31 13:35:21.124321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.115745    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r84bq\" (UniqueName: \"kubernetes.io/projected/7b51cfae-67bd-4df7-a6da-802ec754ffe3-kube-api-access-r84bq\") pod \"payload-processing-e2e-api-a-6aa924-775b66d469-82hlk\" (UID: \"7b51cfae-67bd-4df7-a6da-802ec754ffe3\") " pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"
Oct 31 13:35:21.124321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.116083    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51ddd17-4a94-468b-9ee7-c2f0de9b4528-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p\" (UID: \"b51ddd17-4a94-468b-9ee7-c2f0de9b4528\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"
Oct 31 13:35:21.124321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.116453    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b51cfae-67bd-4df7-a6da-802ec754ffe3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6aa924-775b66d469-82hlk\" (UID: \"7b51cfae-67bd-4df7-a6da-802ec754ffe3\") " pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"
Oct 31 13:35:21.124321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.118759    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51ddd17-4a94-468b-9ee7-c2f0de9b4528-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p\" (UID: \"b51ddd17-4a94-468b-9ee7-c2f0de9b4528\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"
Oct 31 13:35:21.160279 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.141071    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b51cfae-67bd-4df7-a6da-802ec754ffe3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6aa924-775b66d469-82hlk\" (UID: \"7b51cfae-67bd-4df7-a6da-802ec754ffe3\") " pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"
Oct 31 13:35:21.193138 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.176193    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24k24\" (UniqueName: \"kubernetes.io/projected/b51ddd17-4a94-468b-9ee7-c2f0de9b4528-kube-api-access-24k24\") pod \"payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p\" (UID: \"b51ddd17-4a94-468b-9ee7-c2f0de9b4528\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"
Oct 31 13:35:21.267568 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.267538    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"
Oct 31 13:35:21.268839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.268731    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:21.449635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.448566    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r84bq\" (UniqueName: \"kubernetes.io/projected/7b51cfae-67bd-4df7-a6da-802ec754ffe3-kube-api-access-r84bq\") pod \"payload-processing-e2e-api-a-6aa924-775b66d469-82hlk\" (UID: \"7b51cfae-67bd-4df7-a6da-802ec754ffe3\") " pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"
Oct 31 13:35:21.477552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.477174    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"
Oct 31 13:35:21.822994 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.821114    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88" event={"ID":"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e","Type":"ContainerDied","Data":"91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15"}
Oct 31 13:35:21.823319 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.823297    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15"
Oct 31 13:35:21.841582 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.841544    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"
Oct 31 13:35:21.842223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.842201    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:35:21.842778 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.842674    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:21.877907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.876108    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-kube-api-access-xqxld\" (UniqueName: \"kubernetes.io/projected/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-kube-api-access-xqxld\") pod \"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e\" (UID: \"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e\") "
Oct 31 13:35:21.877907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.876367    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-maas-api-xks-ca\") pod \"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e\" (UID: \"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e\") "
Oct 31 13:35:21.877907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.876480    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-maas-api-openshift-ca\") pod \"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e\" (UID: \"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e\") "
Oct 31 13:35:21.947175 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.947107    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-maas-api-openshift-ca" pod "080d9f1e-4bc4-4413-9cf9-80e1b7ac941e" (UID: "080d9f1e-4bc4-4413-9cf9-80e1b7ac941e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:35:21.982920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.965995    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl" event={"ID":"a4853846-205d-4d62-a056-db336dee24cb","Type":"ContainerDied","Data":"2830ff2fb5bd14f0bdacd23016342a74a8344a7a00298f6730fd9f5bcdeb676e"}
Oct 31 13:35:21.983248 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.983230    4428 scope.go:122] "RemoveContainer" containerID="c87a1754d7427fa96dcfc16168934d30dfde33085f41744efb5f8460046f5e5d"
Oct 31 13:35:21.983652 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.982580    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4853846-205d-4d62-a056-db336dee24cb-kube-api-access-w595z\" (UniqueName: \"kubernetes.io/projected/a4853846-205d-4d62-a056-db336dee24cb-kube-api-access-w595z\") pod \"a4853846-205d-4d62-a056-db336dee24cb\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") "
Oct 31 13:35:21.983913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.983892    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-istio-data\") pod \"a4853846-205d-4d62-a056-db336dee24cb\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") "
Oct 31 13:35:21.984189 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.984174    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4853846-205d-4d62-a056-db336dee24cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4853846-205d-4d62-a056-db336dee24cb-istio-podinfo\") pod \"a4853846-205d-4d62-a056-db336dee24cb\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") "
Oct 31 13:35:21.984576 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.984561    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-credential-socket\") pod \"a4853846-205d-4d62-a056-db336dee24cb\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") "
Oct 31 13:35:21.984813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.984800    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4853846-205d-4d62-a056-db336dee24cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4853846-205d-4d62-a056-db336dee24cb-istiod-ca-cert\") pod \"a4853846-205d-4d62-a056-db336dee24cb\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") "
Oct 31 13:35:21.984960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.984947    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-workload-certs\") pod \"a4853846-205d-4d62-a056-db336dee24cb\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") "
Oct 31 13:35:21.985089 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.985078    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-istio-envoy\") pod \"a4853846-205d-4d62-a056-db336dee24cb\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") "
Oct 31 13:35:21.985223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.985212    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4853846-205d-4d62-a056-db336dee24cb-istio-token\" (UniqueName: \"kubernetes.io/projected/a4853846-205d-4d62-a056-db336dee24cb-istio-token\") pod \"a4853846-205d-4d62-a056-db336dee24cb\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") "
Oct 31 13:35:21.985474 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.985462    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-maas-api-openshift-ca\") pod \"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e\" (UID: \"080d9f1e-4bc4-4413-9cf9-80e1b7ac941e\") "
Oct 31 13:35:21.985653 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.985641    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-workload-socket\") pod \"a4853846-205d-4d62-a056-db336dee24cb\" (UID: \"a4853846-205d-4d62-a056-db336dee24cb\") "
Oct 31 13:35:21.992715 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.992691    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-istio-data" pod "a4853846-205d-4d62-a056-db336dee24cb" (UID: "a4853846-205d-4d62-a056-db336dee24cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:35:21.996058 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.996013    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4853846-205d-4d62-a056-db336dee24cb-istiod-ca-cert" pod "a4853846-205d-4d62-a056-db336dee24cb" (UID: "a4853846-205d-4d62-a056-db336dee24cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:35:21.999343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:21.999313    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-credential-socket" pod "a4853846-205d-4d62-a056-db336dee24cb" (UID: "a4853846-205d-4d62-a056-db336dee24cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:35:22.000081 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:35:22.000067    4428 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e/volumes/kubernetes.io~configmap/maas-api-openshift-ca
Oct 31 13:35:22.000281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.000263    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-maas-api-openshift-ca" pod "080d9f1e-4bc4-4413-9cf9-80e1b7ac941e" (UID: "080d9f1e-4bc4-4413-9cf9-80e1b7ac941e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:35:22.008209 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.000831    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-workload-socket" pod "a4853846-205d-4d62-a056-db336dee24cb" (UID: "a4853846-205d-4d62-a056-db336dee24cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:35:22.008350 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.001527    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-workload-certs" pod "a4853846-205d-4d62-a056-db336dee24cb" (UID: "a4853846-205d-4d62-a056-db336dee24cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:35:22.034506 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.034067    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-kube-api-access-xqxld" pod "080d9f1e-4bc4-4413-9cf9-80e1b7ac941e" (UID: "080d9f1e-4bc4-4413-9cf9-80e1b7ac941e"). InnerVolumeSpecName "kube-api-access-xqxld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:35:22.046937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.045239    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-maas-api-xks-ca" pod "080d9f1e-4bc4-4413-9cf9-80e1b7ac941e" (UID: "080d9f1e-4bc4-4413-9cf9-80e1b7ac941e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:35:22.185944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.172426    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqxld\" (UniqueName: \"kubernetes.io/projected/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-kube-api-access-xqxld\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:22.185944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.172474    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:22.185944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.172491    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4853846-205d-4d62-a056-db336dee24cb-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:22.185944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.172506    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:22.185944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.172521    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:22.185944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.172542    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:22.185944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.172557    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:22.185944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.172571    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:22.217248 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.217201    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:22.224775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.224721    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4853846-205d-4d62-a056-db336dee24cb-kube-api-access-w595z" pod "a4853846-205d-4d62-a056-db336dee24cb" (UID: "a4853846-205d-4d62-a056-db336dee24cb"). InnerVolumeSpecName "kube-api-access-w595z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:35:22.240965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.240913    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-istio-envoy" pod "a4853846-205d-4d62-a056-db336dee24cb" (UID: "a4853846-205d-4d62-a056-db336dee24cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:35:22.244947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.243185    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4853846-205d-4d62-a056-db336dee24cb-istio-podinfo" pod "a4853846-205d-4d62-a056-db336dee24cb" (UID: "a4853846-205d-4d62-a056-db336dee24cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:35:22.260071 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.260010    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4853846-205d-4d62-a056-db336dee24cb-istio-token" pod "a4853846-205d-4d62-a056-db336dee24cb" (UID: "a4853846-205d-4d62-a056-db336dee24cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:35:22.300180 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.284288    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4853846-205d-4d62-a056-db336dee24cb-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:22.300432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.300407    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4853846-205d-4d62-a056-db336dee24cb-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:22.300558 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.300544    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w595z\" (UniqueName: \"kubernetes.io/projected/a4853846-205d-4d62-a056-db336dee24cb-kube-api-access-w595z\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:22.300646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.300636    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4853846-205d-4d62-a056-db336dee24cb-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:22.300841 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.300796    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:22.647034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.645482    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:22.663745 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.663710    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"]
Oct 31 13:35:22.726000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.725969    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-393e027a-fcd8l"]
Oct 31 13:35:22.780861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.780823    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="095761dd-4e84-4051-b472-b85d43d444c9" path="/var/lib/kubelet/pods/095761dd-4e84-4051-b472-b85d43d444c9/volumes"
Oct 31 13:35:22.781918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.781853    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e385a07f-5be6-4e14-ba31-3fc5dbbd7de0" path="/var/lib/kubelet/pods/e385a07f-5be6-4e14-ba31-3fc5dbbd7de0/volumes"
Oct 31 13:35:22.927499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:22.923352    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:23.172508 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:23.172464    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"]
Oct 31 13:35:23.359087 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:23.356212    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"
Oct 31 13:35:23.359087 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:23.358407    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"
Oct 31 13:35:23.438946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:23.436751    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"]
Oct 31 13:35:23.576198 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:23.576160    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-p9ncl"]
Oct 31 13:35:24.320957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:24.294611    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:24.331024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:24.302541    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:35:24.431930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:24.418834    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:25.173896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:25.172981    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:25.253952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:25.252071    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:35:25.354205 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:25.354090    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:35:25.378390 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:25.378337    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:35:25.540767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:25.528460    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:25.596950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:25.576553    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:35:25.689928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:25.649069    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:35:25.943956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:25.939234    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p" podStartSLOduration=5.93921155 podStartE2EDuration="5.93921155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:25.254243133 +0000 UTC m=+1922.652591762" watchObservedRunningTime="2026-10-31 13:35:25.93921155 +0000 UTC m=+1923.337560178"
Oct 31 13:35:26.027555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:26.027501    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:26.047034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:26.037773    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4853846-205d-4d62-a056-db336dee24cb" path="/var/lib/kubelet/pods/a4853846-205d-4d62-a056-db336dee24cb/volumes"
Oct 31 13:35:26.064913 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:35:26.057684    4428 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.277s"
Oct 31 13:35:26.064913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:26.057743    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"
Oct 31 13:35:26.064913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:26.057887    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:35:26.064913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:26.057959    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:35:26.064913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:26.057986    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"]
Oct 31 13:35:26.064913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:26.058038    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p" event={"ID":"b51ddd17-4a94-468b-9ee7-c2f0de9b4528","Type":"ContainerStarted","Data":"3877cb11cf92ccf40aece332338eb9cb55aa68c4b487c3decd13f2282cae6004"}
Oct 31 13:35:26.064913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:26.058083    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p" event={"ID":"b51ddd17-4a94-468b-9ee7-c2f0de9b4528","Type":"ContainerStarted","Data":"3843a5ca6a3b79e436863ead75cbb0209e9fda715dfd387bd8a75626712331fe"}
Oct 31 13:35:26.173526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:26.173465    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:26.255057 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:26.255007    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:26.452894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:26.452442    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:26.496906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:26.479561    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:35:26.779720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:26.292846    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:35:26.816199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:26.497295    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:35:26.816605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:26.497354    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:26.816824 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:26.634546    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:35:26.828893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:26.654145    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:27.225036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.222632    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:27.294896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.272660    4428 scope.go:122] "RemoveContainer" containerID="353a7e4925baeed350a790e8fd6e7798c3e6ccab3821b60d961430fa48764968"
Oct 31 13:35:27.334918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.333893    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:27.462892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.456395    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:27.533200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.531145    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:35:27.533200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.531193    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"]
Oct 31 13:35:27.594981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.594942    4428 cpu_manager.go:414] "containerMap: removing container" podUID="080d9f1e-4bc4-4413-9cf9-80e1b7ac941e" containerName="revoke-keys"
Oct 31 13:35:27.596957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.596925    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080d9f1e-4bc4-4413-9cf9-80e1b7ac941e" containerName="revoke-keys"
Oct 31 13:35:27.653096 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.653054    4428 cpu_manager.go:414] "containerMap: removing container" podUID="a4853846-205d-4d62-a056-db336dee24cb" containerName="istio-proxy"
Oct 31 13:35:27.653317 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.653300    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4853846-205d-4d62-a056-db336dee24cb" containerName="istio-proxy"
Oct 31 13:35:27.677989 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.613080    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-8b7s6" podUID="149a9b52-ef95-4569-b947-e6e14d707339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:27.709949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.709921    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="080d9f1e-4bc4-4413-9cf9-80e1b7ac941e" containerName="revoke-keys"
Oct 31 13:35:27.710123 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.710110    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4853846-205d-4d62-a056-db336dee24cb" containerName="istio-proxy"
Oct 31 13:35:27.743236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.743174    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:27.770419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.770370    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:27.848443 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.825989    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:27.849644 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.826148    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:35:27.850099 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.826174    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:35:27.886235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.886202    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"]
Oct 31 13:35:27.929222 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.929187    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:35:27.929537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.929515    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:35:27.929773 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.929761    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:35:28.006729 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.006680    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:28.038891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:27.829453    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:28.038891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.036390    4428 scope.go:122] "RemoveContainer" containerID="a3bdec267cbb1e734c67f68c00a2d1fe31caf1f6e1ce83f2dd885359070fdb69"
Oct 31 13:35:28.093905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.093772    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:28.125005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.122352    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-15dc2f-openshift-default-dockercfg-5tkcp\""
Oct 31 13:35:28.145557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.019552    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk" event={"ID":"7b51cfae-67bd-4df7-a6da-802ec754ffe3","Type":"ContainerStarted","Data":"91c8e021cb302f40fb91fedd9d07b5b36c96aef44a692612a795cb6ade80d4bc"}
Oct 31 13:35:28.223850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.223820    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"
Oct 31 13:35:28.224290 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.224271    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"
Oct 31 13:35:28.224461 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.223717    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:28.358921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.357090    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:28.543073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.454334    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:28.556901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.543364    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:28.565904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.557310    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a90471-68bd-4059-ba8b-8719805293e8-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:28.569903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.569043    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:28.569903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.569095    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:28.569903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.569127    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a90471-68bd-4059-ba8b-8719805293e8-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:28.569903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.569165    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a90471-68bd-4059-ba8b-8719805293e8-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:28.569903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.569203    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9czv\" (UniqueName: \"kubernetes.io/projected/50a90471-68bd-4059-ba8b-8719805293e8-kube-api-access-f9czv\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:28.569903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.569363    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:28.569903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.537715    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:28.613892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.489318    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:28.613892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.528973    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded"
Oct 31 13:35:28.671917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.671848    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:28.672233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.672213    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:28.672494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.672466    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a90471-68bd-4059-ba8b-8719805293e8-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:28.701944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.701905    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:28.761209 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.716384    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:29.011856 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.982997    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a90471-68bd-4059-ba8b-8719805293e8-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:29.176114 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:29.176068    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a90471-68bd-4059-ba8b-8719805293e8-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:29.193072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:29.188084    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9czv\" (UniqueName: \"kubernetes.io/projected/50a90471-68bd-4059-ba8b-8719805293e8-kube-api-access-f9czv\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:29.233522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:29.201467    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:29.233522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.903288    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:29.233522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.905794    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:35:29.233522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.905847    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:35:29.233522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.921745    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Oct 31 13:35:29.233522 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:35:28.982927    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice/crio-91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15\": RecentStats: unable to find data in memory cache]"
Oct 31 13:35:29.233522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:29.224898    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:35:29.242929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:29.235804    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk" podStartSLOduration=9.235780546 podStartE2EDuration="9.235780546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:28.903448941 +0000 UTC m=+1926.301797565" watchObservedRunningTime="2026-10-31 13:35:29.235780546 +0000 UTC m=+1926.634129171"
Oct 31 13:35:29.276947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:29.243772    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:29.387325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:29.098834    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a90471-68bd-4059-ba8b-8719805293e8-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:29.387575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:29.149426    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:29.387681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:29.151538    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:35:29.395959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.716304    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:29.396278 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:29.182588    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a90471-68bd-4059-ba8b-8719805293e8-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:29.396422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.807578    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:29.396522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.792959    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:29.400900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:28.905759    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:35:29.401575 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:35:29.009201    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice/crio-91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:35:29.464280 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:29.063925    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:29.508085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:29.283116    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a90471-68bd-4059-ba8b-8719805293e8-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:29.689605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:29.689559    4428 scope.go:122] "RemoveContainer" containerID="77b85e79d4a3f997795bdf59ebc8b2614da484bad221a20b5ebce63e63558ff3"
Oct 31 13:35:29.899952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:29.896276    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:35:29.915296 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:29.915249    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:35:30.035213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:30.032781    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9czv\" (UniqueName: \"kubernetes.io/projected/50a90471-68bd-4059-ba8b-8719805293e8-kube-api-access-f9czv\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:30.201079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:30.187002    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:30.204202 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:30.204143    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:30.228631 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:30.228588    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:30.240548 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:30.231343    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:35:30.347898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:30.343699    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:35:30.392894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:30.392126    4428 scope.go:122] "RemoveContainer" containerID="a60c43c04dfd83da1843b3a6429d24d45f6f65182e1551e9a8026d708bd1e507"
Oct 31 13:35:30.404101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:30.404046    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47" podUID="ea978050-20b0-4148-ab41-75844fe1b637" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:30.520038 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:30.519984    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:35:30.751516 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:30.751464    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk" event={"ID":"7b51cfae-67bd-4df7-a6da-802ec754ffe3","Type":"ContainerStarted","Data":"94332cf83a4ad7e82b24fac4c62055fa9faa0a291eb1f4662b0a8fea8cb9e18f"}
Oct 31 13:35:30.762469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:30.538339    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:30.809128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:30.778094    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:35:30.839089 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:30.811805    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:30.989895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:30.981526    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:31.385905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:31.385279    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:31.442940 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:31.441197    4428 scope.go:122] "RemoveContainer" containerID="e643d2c2f076ad3ace7e1b7b780dc17597394c91bfc0ce6b9889f3809d48eb67"
Oct 31 13:35:31.444798 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:31.444760    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:35:31.514251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:31.513148    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:31.524768 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:31.524721    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:31.539676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:31.537901    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:35:31.539676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:31.537945    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:35:31.539676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:31.537990    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:35:31.539676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:31.538029    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:35:31.539676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:31.538062    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:35:31.626308 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:31.626281    4428 scope.go:122] "RemoveContainer" containerID="3c1d7a16430b3e57e822b84e77ca074406aec9ce0308124b76650db09c795e32"
Oct 31 13:35:31.903975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:31.903952    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"
Oct 31 13:35:31.928689 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:31.928663    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"]
Oct 31 13:35:31.933514 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:31.933475    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:35:31.933633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:31.933563    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:35:31.933633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:31.933608    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:35:32.326583 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:32.326498    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:32.910177 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:32.910137    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt" event={"ID":"50a90471-68bd-4059-ba8b-8719805293e8","Type":"ContainerStarted","Data":"ebaf9521a92268d7cd75b8bbaaf944dd891d705d2373851381286fec4dcb6e5d"}
Oct 31 13:35:32.910668 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:32.910201    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt" event={"ID":"50a90471-68bd-4059-ba8b-8719805293e8","Type":"ContainerStarted","Data":"833ef678252324648d4ef3e7dacff7c34ec78b42a4b4e212a547beb83debcb93"}
Oct 31 13:35:32.945982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:32.945917    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt" podStartSLOduration=6.945897593 podStartE2EDuration="6.945897593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:32.938276521 +0000 UTC m=+1930.336625150" watchObservedRunningTime="2026-10-31 13:35:32.945897593 +0000 UTC m=+1930.344246218"
Oct 31 13:35:32.984255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:32.984217    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:33.119940 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:33.115384    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:33.490899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:33.489187    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:33.991619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:33.991515    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:34.109268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:34.089054    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:34.256954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:34.251820    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:34.585950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:34.582512    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:34.994081 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:34.975446    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:34.994081 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:34.975976    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:34.996234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:34.995523    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:35:35.385855 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:35.385744    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:35.600051 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:35.598894    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:36.313320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:36.313231    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:36.353284 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:36.353246    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"]
Oct 31 13:35:36.360354 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:36.360321    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-466002-6v6zq"]
Oct 31 13:35:36.472314 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:36.472282    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:36.630795 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:36.630709    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:36.771913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:36.771850    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="455a838e-c5ee-460e-80db-c8b3bf879c28" path="/var/lib/kubelet/pods/455a838e-c5ee-460e-80db-c8b3bf879c28/volumes"
Oct 31 13:35:36.855375 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:36.855138    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"]
Oct 31 13:35:36.873317 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:36.865811    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:36.901078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:36.900993    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"]
Oct 31 13:35:36.901345 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:36.901144    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"
Oct 31 13:35:36.907338 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:36.907269    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-15dc2f-dockercfg-sp55z\""
Oct 31 13:35:36.945682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:36.945656    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-15dc2f\""
Oct 31 13:35:36.986070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:36.986029    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4ltl\" (UniqueName: \"kubernetes.io/projected/2cba5737-a6a1-4c1c-8b1d-24f1f7162a60-kube-api-access-r4ltl\") pod \"payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v\" (UID: \"2cba5737-a6a1-4c1c-8b1d-24f1f7162a60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"
Oct 31 13:35:36.986509 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:36.986480    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cba5737-a6a1-4c1c-8b1d-24f1f7162a60-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v\" (UID: \"2cba5737-a6a1-4c1c-8b1d-24f1f7162a60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"
Oct 31 13:35:37.097097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:37.095211    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:37.107916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:37.105965    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4ltl\" (UniqueName: \"kubernetes.io/projected/2cba5737-a6a1-4c1c-8b1d-24f1f7162a60-kube-api-access-r4ltl\") pod \"payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v\" (UID: \"2cba5737-a6a1-4c1c-8b1d-24f1f7162a60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"
Oct 31 13:35:37.111650 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:37.111620    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cba5737-a6a1-4c1c-8b1d-24f1f7162a60-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v\" (UID: \"2cba5737-a6a1-4c1c-8b1d-24f1f7162a60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"
Oct 31 13:35:37.121960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:37.121611    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cba5737-a6a1-4c1c-8b1d-24f1f7162a60-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v\" (UID: \"2cba5737-a6a1-4c1c-8b1d-24f1f7162a60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"
Oct 31 13:35:37.219466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:37.207956    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4ltl\" (UniqueName: \"kubernetes.io/projected/2cba5737-a6a1-4c1c-8b1d-24f1f7162a60-kube-api-access-r4ltl\") pod \"payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v\" (UID: \"2cba5737-a6a1-4c1c-8b1d-24f1f7162a60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"
Oct 31 13:35:37.265917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:37.265331    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"
Oct 31 13:35:37.826960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:37.826931    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"]
Oct 31 13:35:37.827398 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:35:37.827370    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cba5737_a6a1_4c1c_8b1d_24f1f7162a60.slice/crio-5fec98c22105d49a94629aa2522496cb09fb0988bad34016b4fed208e34ee011 WatchSource:0}: Error finding container 5fec98c22105d49a94629aa2522496cb09fb0988bad34016b4fed208e34ee011: Status 404 returned error can't find the container with id 5fec98c22105d49a94629aa2522496cb09fb0988bad34016b4fed208e34ee011
Oct 31 13:35:38.185476 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:38.181861    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v" event={"ID":"2cba5737-a6a1-4c1c-8b1d-24f1f7162a60","Type":"ContainerStarted","Data":"41b23ae38f5092c43300be2a3b088fb8d20417efce971f7d1772bd09bbcb3219"}
Oct 31 13:35:38.185476 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:38.182009    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v" event={"ID":"2cba5737-a6a1-4c1c-8b1d-24f1f7162a60","Type":"ContainerStarted","Data":"5fec98c22105d49a94629aa2522496cb09fb0988bad34016b4fed208e34ee011"}
Oct 31 13:35:38.185476 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:38.183335    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"
Oct 31 13:35:38.216539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:38.216468    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v" podStartSLOduration=2.216444237 podStartE2EDuration="2.216444237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:38.213009548 +0000 UTC m=+1935.611358173" watchObservedRunningTime="2026-10-31 13:35:38.216444237 +0000 UTC m=+1935.614792862"
Oct 31 13:35:38.223461 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:38.221500    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:38.258390 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:38.258031    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:35:38.258390 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:38.258102    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n" podUID="b9fb27a2-762a-4256-8c53-89a6bb50e654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:38.395926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:38.387466    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:39.649963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:39.649933    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:39.748349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:39.748268    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"]
Oct 31 13:35:39.749595 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:35:39.749561    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice/crio-91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:35:39.766722 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:39.766680    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"]
Oct 31 13:35:39.766862 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:39.766808    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"
Oct 31 13:35:39.783440 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:39.783413    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cbb3d5-dockercfg-vxxvs\""
Oct 31 13:35:39.814250 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:39.814215    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/093cde47-6a66-4a54-8ee4-6c205794b2f4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k\" (UID: \"093cde47-6a66-4a54-8ee4-6c205794b2f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"
Oct 31 13:35:39.814423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:39.814381    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xcnt\" (UniqueName: \"kubernetes.io/projected/093cde47-6a66-4a54-8ee4-6c205794b2f4-kube-api-access-6xcnt\") pod \"payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k\" (UID: \"093cde47-6a66-4a54-8ee4-6c205794b2f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"
Oct 31 13:35:39.857900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:39.823447    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:39.857900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:39.843267    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cbb3d5\""
Oct 31 13:35:39.967018 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:39.918506    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/093cde47-6a66-4a54-8ee4-6c205794b2f4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k\" (UID: \"093cde47-6a66-4a54-8ee4-6c205794b2f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"
Oct 31 13:35:39.967364 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:39.967338    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xcnt\" (UniqueName: \"kubernetes.io/projected/093cde47-6a66-4a54-8ee4-6c205794b2f4-kube-api-access-6xcnt\") pod \"payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k\" (UID: \"093cde47-6a66-4a54-8ee4-6c205794b2f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"
Oct 31 13:35:39.968258 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:39.965851    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/093cde47-6a66-4a54-8ee4-6c205794b2f4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k\" (UID: \"093cde47-6a66-4a54-8ee4-6c205794b2f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"
Oct 31 13:35:40.054894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:40.054187    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xcnt\" (UniqueName: \"kubernetes.io/projected/093cde47-6a66-4a54-8ee4-6c205794b2f4-kube-api-access-6xcnt\") pod \"payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k\" (UID: \"093cde47-6a66-4a54-8ee4-6c205794b2f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"
Oct 31 13:35:40.102764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:40.102717    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"
Oct 31 13:35:40.388988 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:35:40.388937    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice/crio-91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15\": RecentStats: unable to find data in memory cache]"
Oct 31 13:35:40.553004 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:40.552976    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"]
Oct 31 13:35:40.578629 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:35:40.578603    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093cde47_6a66_4a54_8ee4_6c205794b2f4.slice/crio-5ce1bcf512f62f140df260e3d3f7110003e448e619b506049e6c095229e01a8c WatchSource:0}: Error finding container 5ce1bcf512f62f140df260e3d3f7110003e448e619b506049e6c095229e01a8c: Status 404 returned error can't find the container with id 5ce1bcf512f62f140df260e3d3f7110003e448e619b506049e6c095229e01a8c
Oct 31 13:35:41.124732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.124461    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:41.245897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.241198    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k" event={"ID":"093cde47-6a66-4a54-8ee4-6c205794b2f4","Type":"ContainerStarted","Data":"db01943d22177f5c12f5bffb8fe8344171ac2d74639aed03fbc60dffc93e2db0"}
Oct 31 13:35:41.245897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.241267    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k" event={"ID":"093cde47-6a66-4a54-8ee4-6c205794b2f4","Type":"ContainerStarted","Data":"5ce1bcf512f62f140df260e3d3f7110003e448e619b506049e6c095229e01a8c"}
Oct 31 13:35:41.245897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.242741    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"
Oct 31 13:35:41.245897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.245264    4428 generic.go:356] "Generic (PLEG): container finished" podID="b9fb27a2-762a-4256-8c53-89a6bb50e654" containerID="8b2d78d52433b2821008621c03483af3a2af47d779c95e2c2a80f872e1a77d0e" exitCode=0
Oct 31 13:35:41.245897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.245633    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n" event={"ID":"b9fb27a2-762a-4256-8c53-89a6bb50e654","Type":"ContainerDied","Data":"8b2d78d52433b2821008621c03483af3a2af47d779c95e2c2a80f872e1a77d0e"}
Oct 31 13:35:41.246320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.246283    4428 scope.go:122] "RemoveContainer" containerID="8b2d78d52433b2821008621c03483af3a2af47d779c95e2c2a80f872e1a77d0e"
Oct 31 13:35:41.253406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.251999    4428 generic.go:356] "Generic (PLEG): container finished" podID="d3179e22-1592-44a5-8193-8e7e9aee4b5e" containerID="cbb534617b29013f4c84e26ed1b7c2bf413c252d966860f002d75069f26b5e10" exitCode=0
Oct 31 13:35:41.253406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.252053    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw" event={"ID":"d3179e22-1592-44a5-8193-8e7e9aee4b5e","Type":"ContainerDied","Data":"cbb534617b29013f4c84e26ed1b7c2bf413c252d966860f002d75069f26b5e10"}
Oct 31 13:35:41.253406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.252596    4428 scope.go:122] "RemoveContainer" containerID="cbb534617b29013f4c84e26ed1b7c2bf413c252d966860f002d75069f26b5e10"
Oct 31 13:35:41.256312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.255841    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:35:41.256312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.255981    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:35:41.256312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.256040    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:35:41.284540 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.284472    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k" podStartSLOduration=2.284451557 podStartE2EDuration="2.284451557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:41.266135019 +0000 UTC m=+1938.664483645" watchObservedRunningTime="2026-10-31 13:35:41.284451557 +0000 UTC m=+1938.682800185"
Oct 31 13:35:41.289284 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.289247    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:35:41.289420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.289344    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:35:41.289420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.289391    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:35:41.924994 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:41.924959    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:35:42.310216 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:42.310172    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n" event={"ID":"b9fb27a2-762a-4256-8c53-89a6bb50e654","Type":"ContainerStarted","Data":"9ffddf3cb81a87a3a21b322f62192d392287e6938ab5825ed15d2eae06344e3a"}
Oct 31 13:35:42.314731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:42.314665    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw" event={"ID":"d3179e22-1592-44a5-8193-8e7e9aee4b5e","Type":"ContainerStarted","Data":"dae3cc317ff5a8055e68f2c6b703a6020bf92eb12800d0f5f9e4cd35b4f4d72e"}
Oct 31 13:35:42.364382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:42.364344    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:42.377492 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:42.377455    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:35:42.378754 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:42.378493    4428 patch_prober.go:29] interesting pod/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 13:35:42.378936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:42.378774    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw" podUID="d3179e22-1592-44a5-8193-8e7e9aee4b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 13:35:43.085895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:43.085332    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:35:43.085895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:43.085474    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Oct 31 13:35:43.085895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:43.085517    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n" podUID="b9fb27a2-762a-4256-8c53-89a6bb50e654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Oct 31 13:35:43.388956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:43.388412    4428 patch_prober.go:29] interesting pod/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 13:35:43.388956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:43.388475    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw" podUID="d3179e22-1592-44a5-8193-8e7e9aee4b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 13:35:43.715942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:43.685484    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:43.886895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:43.886456    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:44.189186 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:44.189034    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:44.306891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:44.291458    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Oct 31 13:35:44.306891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:44.291556    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n" podUID="b9fb27a2-762a-4256-8c53-89a6bb50e654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Oct 31 13:35:44.379738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:44.379001    4428 patch_prober.go:29] interesting pod/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 13:35:44.379738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:44.379063    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw" podUID="d3179e22-1592-44a5-8193-8e7e9aee4b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 13:35:44.413142 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:44.413106    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:44.899147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:44.812090    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:44.899147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:44.822992    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:45.087900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.084455    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Oct 31 13:35:45.087900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.084534    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n" podUID="b9fb27a2-762a-4256-8c53-89a6bb50e654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Oct 31 13:35:45.190947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.187100    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"]
Oct 31 13:35:45.396785 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.374395    4428 patch_prober.go:29] interesting pod/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 13:35:45.396785 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.374475    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw" podUID="d3179e22-1592-44a5-8193-8e7e9aee4b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 13:35:45.396785 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.387383    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:45.452778 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.439924    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:45.504898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.479221    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Oct 31 13:35:45.504898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.483172    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:35:45.504898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.491138    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:35:45.536551 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.536517    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"]
Oct 31 13:35:45.536741 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.536657    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"
Oct 31 13:35:45.544249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.544220    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:35:45.544422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.544340    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:35:45.676295 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.676243    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-257dj\" (UniqueName: \"kubernetes.io/projected/d18690c6-12a1-4bdf-a659-33d883ab758f-kube-api-access-257dj\") pod \"maas-api-revoke-keys-e2e-api-b-752869-xl89g\" (UID: \"d18690c6-12a1-4bdf-a659-33d883ab758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"
Oct 31 13:35:45.681143 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.681106    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d18690c6-12a1-4bdf-a659-33d883ab758f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-752869-xl89g\" (UID: \"d18690c6-12a1-4bdf-a659-33d883ab758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"
Oct 31 13:35:45.681641 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.681603    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d18690c6-12a1-4bdf-a659-33d883ab758f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-752869-xl89g\" (UID: \"d18690c6-12a1-4bdf-a659-33d883ab758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"
Oct 31 13:35:45.786280 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.786234    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-257dj\" (UniqueName: \"kubernetes.io/projected/d18690c6-12a1-4bdf-a659-33d883ab758f-kube-api-access-257dj\") pod \"maas-api-revoke-keys-e2e-api-b-752869-xl89g\" (UID: \"d18690c6-12a1-4bdf-a659-33d883ab758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"
Oct 31 13:35:45.786672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.786646    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d18690c6-12a1-4bdf-a659-33d883ab758f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-752869-xl89g\" (UID: \"d18690c6-12a1-4bdf-a659-33d883ab758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"
Oct 31 13:35:45.790899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.786982    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d18690c6-12a1-4bdf-a659-33d883ab758f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-752869-xl89g\" (UID: \"d18690c6-12a1-4bdf-a659-33d883ab758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"
Oct 31 13:35:45.791748 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.791358    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d18690c6-12a1-4bdf-a659-33d883ab758f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-752869-xl89g\" (UID: \"d18690c6-12a1-4bdf-a659-33d883ab758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"
Oct 31 13:35:45.806485 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.806447    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d18690c6-12a1-4bdf-a659-33d883ab758f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-752869-xl89g\" (UID: \"d18690c6-12a1-4bdf-a659-33d883ab758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"
Oct 31 13:35:45.830736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.824711    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:45.865640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.865603    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-257dj\" (UniqueName: \"kubernetes.io/projected/d18690c6-12a1-4bdf-a659-33d883ab758f-kube-api-access-257dj\") pod \"maas-api-revoke-keys-e2e-api-b-752869-xl89g\" (UID: \"d18690c6-12a1-4bdf-a659-33d883ab758f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"
Oct 31 13:35:45.895797 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:45.895751    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"
Oct 31 13:35:46.130080 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:46.128066    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Oct 31 13:35:46.130080 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:46.128150    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n" podUID="b9fb27a2-762a-4256-8c53-89a6bb50e654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Oct 31 13:35:46.396947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:46.396632    4428 patch_prober.go:29] interesting pod/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 13:35:46.396947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:46.396710    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw" podUID="d3179e22-1592-44a5-8193-8e7e9aee4b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 13:35:46.502073 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:35:46.502045    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18690c6_12a1_4bdf_a659_33d883ab758f.slice/crio-5c2ecfd0eedd663182adc3ddac2273ee0fe2878c1cb6895ca311fb952094c7a4 WatchSource:0}: Error finding container 5c2ecfd0eedd663182adc3ddac2273ee0fe2878c1cb6895ca311fb952094c7a4: Status 404 returned error can't find the container with id 5c2ecfd0eedd663182adc3ddac2273ee0fe2878c1cb6895ca311fb952094c7a4
Oct 31 13:35:46.505757 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:46.505731    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"]
Oct 31 13:35:46.526801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:46.526774    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g" event={"ID":"d18690c6-12a1-4bdf-a659-33d883ab758f","Type":"ContainerStarted","Data":"5c2ecfd0eedd663182adc3ddac2273ee0fe2878c1cb6895ca311fb952094c7a4"}
Oct 31 13:35:46.822500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:46.822464    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:47.076915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:47.076804    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Oct 31 13:35:47.077158 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:47.077129    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n" podUID="b9fb27a2-762a-4256-8c53-89a6bb50e654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Oct 31 13:35:47.375947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:47.375320    4428 patch_prober.go:29] interesting pod/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 13:35:47.375947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:47.375382    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw" podUID="d3179e22-1592-44a5-8193-8e7e9aee4b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 13:35:47.537663 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:47.537627    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g" event={"ID":"d18690c6-12a1-4bdf-a659-33d883ab758f","Type":"ContainerStarted","Data":"ebce4d901a51e43baa4261935975c8b59f6133b9c466ea47f91d03138ee7f36d"}
Oct 31 13:35:47.605254 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:47.605183    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g" podStartSLOduration=3.605159906 podStartE2EDuration="3.605159906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:47.601485314 +0000 UTC m=+1944.999833939" watchObservedRunningTime="2026-10-31 13:35:47.605159906 +0000 UTC m=+1945.003508593"
Oct 31 13:35:47.630729 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:47.630640    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:48.080903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:48.080283    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Oct 31 13:35:48.080903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:48.080347    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n" podUID="b9fb27a2-762a-4256-8c53-89a6bb50e654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Oct 31 13:35:48.400016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:48.377263    4428 patch_prober.go:29] interesting pod/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 13:35:48.400016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:48.377331    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw" podUID="d3179e22-1592-44a5-8193-8e7e9aee4b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 13:35:48.619894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:48.605590    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:48.636894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:48.635163    4428 generic.go:356] "Generic (PLEG): container finished" podID="d18690c6-12a1-4bdf-a659-33d883ab758f" containerID="ebce4d901a51e43baa4261935975c8b59f6133b9c466ea47f91d03138ee7f36d" exitCode=0
Oct 31 13:35:48.636894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:48.635450    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g" event={"ID":"d18690c6-12a1-4bdf-a659-33d883ab758f","Type":"ContainerDied","Data":"ebce4d901a51e43baa4261935975c8b59f6133b9c466ea47f91d03138ee7f36d"}
Oct 31 13:35:48.689009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:48.684728    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"]
Oct 31 13:35:48.883902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:48.879045    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"
Oct 31 13:35:49.022406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.022375    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"]
Oct 31 13:35:49.041188 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.041155    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f25w\" (UniqueName: \"kubernetes.io/projected/4d136cbb-2350-444a-82c3-5b595212315f-kube-api-access-7f25w\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh\" (UID: \"4d136cbb-2350-444a-82c3-5b595212315f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"
Oct 31 13:35:49.041349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.041339    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d136cbb-2350-444a-82c3-5b595212315f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh\" (UID: \"4d136cbb-2350-444a-82c3-5b595212315f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"
Oct 31 13:35:49.041439 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.041427    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d136cbb-2350-444a-82c3-5b595212315f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh\" (UID: \"4d136cbb-2350-444a-82c3-5b595212315f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"
Oct 31 13:35:49.129643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.129191    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Oct 31 13:35:49.129643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.129261    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n" podUID="b9fb27a2-762a-4256-8c53-89a6bb50e654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Oct 31 13:35:49.129643 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:35:49.129433    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:35:49.142779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.142751    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d136cbb-2350-444a-82c3-5b595212315f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh\" (UID: \"4d136cbb-2350-444a-82c3-5b595212315f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"
Oct 31 13:35:49.142990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.142906    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d136cbb-2350-444a-82c3-5b595212315f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh\" (UID: \"4d136cbb-2350-444a-82c3-5b595212315f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"
Oct 31 13:35:49.143044 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.143018    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f25w\" (UniqueName: \"kubernetes.io/projected/4d136cbb-2350-444a-82c3-5b595212315f-kube-api-access-7f25w\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh\" (UID: \"4d136cbb-2350-444a-82c3-5b595212315f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"
Oct 31 13:35:49.144168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.144146    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d136cbb-2350-444a-82c3-5b595212315f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh\" (UID: \"4d136cbb-2350-444a-82c3-5b595212315f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"
Oct 31 13:35:49.190968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.187372    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d136cbb-2350-444a-82c3-5b595212315f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh\" (UID: \"4d136cbb-2350-444a-82c3-5b595212315f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"
Oct 31 13:35:49.196633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.194587    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f25w\" (UniqueName: \"kubernetes.io/projected/4d136cbb-2350-444a-82c3-5b595212315f-kube-api-access-7f25w\") pod \"maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh\" (UID: \"4d136cbb-2350-444a-82c3-5b595212315f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"
Oct 31 13:35:49.335158 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.335058    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"
Oct 31 13:35:49.505629 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.503486    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:49.573549 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.573287    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:35:49.582949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.576329    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:35:49.648937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.648839    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:35:49.884603 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.884563    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"]
Oct 31 13:35:49.899744 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:35:49.899681    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice/crio-91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15\": RecentStats: unable to find data in memory cache]"
Oct 31 13:35:49.952624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:49.952600    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"
Oct 31 13:35:50.029242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.029210    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"]
Oct 31 13:35:50.034150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.034116    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2123aec9-vnc9h"]
Oct 31 13:35:50.050847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.050823    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d18690c6-12a1-4bdf-a659-33d883ab758f-kube-api-access-257dj\" (UniqueName: \"kubernetes.io/projected/d18690c6-12a1-4bdf-a659-33d883ab758f-kube-api-access-257dj\") pod \"d18690c6-12a1-4bdf-a659-33d883ab758f\" (UID: \"d18690c6-12a1-4bdf-a659-33d883ab758f\") "
Oct 31 13:35:50.051015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.051001    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d18690c6-12a1-4bdf-a659-33d883ab758f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d18690c6-12a1-4bdf-a659-33d883ab758f-maas-api-openshift-ca\") pod \"d18690c6-12a1-4bdf-a659-33d883ab758f\" (UID: \"d18690c6-12a1-4bdf-a659-33d883ab758f\") "
Oct 31 13:35:50.051218 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.051200    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d18690c6-12a1-4bdf-a659-33d883ab758f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d18690c6-12a1-4bdf-a659-33d883ab758f-maas-api-xks-ca\") pod \"d18690c6-12a1-4bdf-a659-33d883ab758f\" (UID: \"d18690c6-12a1-4bdf-a659-33d883ab758f\") "
Oct 31 13:35:50.051393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.051365    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d18690c6-12a1-4bdf-a659-33d883ab758f-maas-api-openshift-ca" pod "d18690c6-12a1-4bdf-a659-33d883ab758f" (UID: "d18690c6-12a1-4bdf-a659-33d883ab758f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:35:50.052121 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.052098    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d18690c6-12a1-4bdf-a659-33d883ab758f-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:50.053301 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.053276    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18690c6-12a1-4bdf-a659-33d883ab758f-kube-api-access-257dj" pod "d18690c6-12a1-4bdf-a659-33d883ab758f" (UID: "d18690c6-12a1-4bdf-a659-33d883ab758f"). InnerVolumeSpecName "kube-api-access-257dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:35:50.053369 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.053304    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d18690c6-12a1-4bdf-a659-33d883ab758f-maas-api-xks-ca" pod "d18690c6-12a1-4bdf-a659-33d883ab758f" (UID: "d18690c6-12a1-4bdf-a659-33d883ab758f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:35:50.081504 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.081476    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:35:50.081914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.081887    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:35:50.082692 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.082667    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 13:35:50.154382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.154283    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-257dj\" (UniqueName: \"kubernetes.io/projected/d18690c6-12a1-4bdf-a659-33d883ab758f-kube-api-access-257dj\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:50.154382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.154316    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d18690c6-12a1-4bdf-a659-33d883ab758f-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:50.218639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.218616    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"
Oct 31 13:35:50.758503 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.757941    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"
Oct 31 13:35:50.767077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.767046    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f726de20-47fe-4518-b6b9-cdd3a3d579e3" path="/var/lib/kubelet/pods/f726de20-47fe-4518-b6b9-cdd3a3d579e3/volumes"
Oct 31 13:35:50.767566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.767545    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g" event={"ID":"d18690c6-12a1-4bdf-a659-33d883ab758f","Type":"ContainerDied","Data":"5c2ecfd0eedd663182adc3ddac2273ee0fe2878c1cb6895ca311fb952094c7a4"}
Oct 31 13:35:50.767627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.767585    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c2ecfd0eedd663182adc3ddac2273ee0fe2878c1cb6895ca311fb952094c7a4"
Oct 31 13:35:50.767627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.767595    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh" event={"ID":"4d136cbb-2350-444a-82c3-5b595212315f","Type":"ContainerStarted","Data":"5f4aa9fa07437bbb943eb2a2762a625bc29e635735b6ee20dbf03405950d7794"}
Oct 31 13:35:50.767744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.767630    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh" event={"ID":"4d136cbb-2350-444a-82c3-5b595212315f","Type":"ContainerStarted","Data":"6b5eb9cc4cb38a6ffb1059319a975a1af64e521877785c747bdd787a3d241e27"}
Oct 31 13:35:50.801180 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:50.801142    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:51.547898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:51.546972    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:51.770438 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:51.770402    4428 generic.go:356] "Generic (PLEG): container finished" podID="4d136cbb-2350-444a-82c3-5b595212315f" containerID="5f4aa9fa07437bbb943eb2a2762a625bc29e635735b6ee20dbf03405950d7794" exitCode=0
Oct 31 13:35:51.770895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:51.770478    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh" event={"ID":"4d136cbb-2350-444a-82c3-5b595212315f","Type":"ContainerDied","Data":"5f4aa9fa07437bbb943eb2a2762a625bc29e635735b6ee20dbf03405950d7794"}
Oct 31 13:35:52.338222 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:52.338174    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:52.944728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:52.944697    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"
Oct 31 13:35:53.082486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:53.082407    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d136cbb-2350-444a-82c3-5b595212315f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d136cbb-2350-444a-82c3-5b595212315f-maas-api-xks-ca\") pod \"4d136cbb-2350-444a-82c3-5b595212315f\" (UID: \"4d136cbb-2350-444a-82c3-5b595212315f\") "
Oct 31 13:35:53.082486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:53.082464    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d136cbb-2350-444a-82c3-5b595212315f-kube-api-access-7f25w\" (UniqueName: \"kubernetes.io/projected/4d136cbb-2350-444a-82c3-5b595212315f-kube-api-access-7f25w\") pod \"4d136cbb-2350-444a-82c3-5b595212315f\" (UID: \"4d136cbb-2350-444a-82c3-5b595212315f\") "
Oct 31 13:35:53.082670 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:53.082632    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d136cbb-2350-444a-82c3-5b595212315f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d136cbb-2350-444a-82c3-5b595212315f-maas-api-openshift-ca\") pod \"4d136cbb-2350-444a-82c3-5b595212315f\" (UID: \"4d136cbb-2350-444a-82c3-5b595212315f\") "
Oct 31 13:35:53.083041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:53.083011    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d136cbb-2350-444a-82c3-5b595212315f-maas-api-openshift-ca" pod "4d136cbb-2350-444a-82c3-5b595212315f" (UID: "4d136cbb-2350-444a-82c3-5b595212315f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:35:53.083407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:53.083388    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d136cbb-2350-444a-82c3-5b595212315f-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:53.084622 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:53.084600    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d136cbb-2350-444a-82c3-5b595212315f-maas-api-xks-ca" pod "4d136cbb-2350-444a-82c3-5b595212315f" (UID: "4d136cbb-2350-444a-82c3-5b595212315f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:35:53.084727 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:53.084632    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d136cbb-2350-444a-82c3-5b595212315f-kube-api-access-7f25w" pod "4d136cbb-2350-444a-82c3-5b595212315f" (UID: "4d136cbb-2350-444a-82c3-5b595212315f"). InnerVolumeSpecName "kube-api-access-7f25w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:35:53.159897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:53.157101    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:53.187137 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:53.186027    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d136cbb-2350-444a-82c3-5b595212315f-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:53.187137 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:53.186068    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f25w\" (UniqueName: \"kubernetes.io/projected/4d136cbb-2350-444a-82c3-5b595212315f-kube-api-access-7f25w\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:35:53.324972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:53.324940    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"
Oct 31 13:35:53.789393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:53.789364    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"
Oct 31 13:35:53.789552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:53.789401    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh" event={"ID":"4d136cbb-2350-444a-82c3-5b595212315f","Type":"ContainerDied","Data":"6b5eb9cc4cb38a6ffb1059319a975a1af64e521877785c747bdd787a3d241e27"}
Oct 31 13:35:53.789552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:53.789447    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b5eb9cc4cb38a6ffb1059319a975a1af64e521877785c747bdd787a3d241e27"
Oct 31 13:35:53.958890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:53.958509    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:54.035424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:54.035387    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"]
Oct 31 13:35:54.040465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:54.040397    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67f6fbe5-txghh"]
Oct 31 13:35:54.679031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:54.678994    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:54.779276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:54.776281    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54553f5b-58e8-45da-a20c-bdd60e1dcfd6" path="/var/lib/kubelet/pods/54553f5b-58e8-45da-a20c-bdd60e1dcfd6/volumes"
Oct 31 13:35:54.842701 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:54.842668    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"]
Oct 31 13:35:54.850298 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:54.850265    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"]
Oct 31 13:35:54.979918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:54.979860    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:55.193891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:55.189037    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:55.699798 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:55.699766    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:55.892889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:55.890290    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:55.970857 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:35:55.970824    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice/crio-91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:35:56.811890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:56.810173    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:57.607311 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:57.607282    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:57.774508 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:57.774458    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:58.685823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:58.685790    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"]
Oct 31 13:35:58.780381 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:58.780347    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:58.941084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:58.941002    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:59.727520 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:59.727476    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:35:59.854366 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:59.854270    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml" podUID="9162cb66-9fcd-4fa4-89d2-f7044b8870a4" containerName="payload-pre-processing" containerID="cri-o://f9dabf4f694b7e704015f5086384a9a623c2a0c749c72a6ad58c181011ab947b" gracePeriod=25
Oct 31 13:35:59.857793 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:59.856284    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b" podUID="8baa85c2-ada3-42ca-849c-afffc3e1c13e" containerName="payload-processing" containerID="cri-o://976014cb385c6e29d29f15c4ef4ccc130e285dd7760e134bbe835b2b582e1ff4" gracePeriod=25
Oct 31 13:35:59.915717 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:35:59.915678    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice/crio-91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15\": RecentStats: unable to find data in memory cache]"
Oct 31 13:35:59.957006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:59.956977    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"]
Oct 31 13:35:59.957784 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:59.957770    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d18690c6-12a1-4bdf-a659-33d883ab758f" containerName="revoke-keys"
Oct 31 13:35:59.957854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:59.957785    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d18690c6-12a1-4bdf-a659-33d883ab758f" containerName="revoke-keys"
Oct 31 13:35:59.957854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:59.957813    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4d136cbb-2350-444a-82c3-5b595212315f" containerName="revoke-keys"
Oct 31 13:35:59.957854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:59.957818    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d136cbb-2350-444a-82c3-5b595212315f" containerName="revoke-keys"
Oct 31 13:35:59.957982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:59.957970    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d136cbb-2350-444a-82c3-5b595212315f" containerName="revoke-keys"
Oct 31 13:35:59.958014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:59.957995    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d18690c6-12a1-4bdf-a659-33d883ab758f" containerName="revoke-keys"
Oct 31 13:35:59.962276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:59.962250    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"
Oct 31 13:35:59.967198 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:59.967172    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:35:59.967822 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:59.967604    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:35:59.976915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:59.973150    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"]
Oct 31 13:35:59.991980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:59.991948    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk67d\" (UniqueName: \"kubernetes.io/projected/9c92c3cd-9053-4a35-925c-939e82694732-kube-api-access-dk67d\") pod \"maas-api-revoke-sub-41939f60-hbqs4\" (UID: \"9c92c3cd-9053-4a35-925c-939e82694732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"
Oct 31 13:35:59.992108 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:59.992016    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c92c3cd-9053-4a35-925c-939e82694732-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41939f60-hbqs4\" (UID: \"9c92c3cd-9053-4a35-925c-939e82694732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"
Oct 31 13:35:59.992191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:35:59.992130    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c92c3cd-9053-4a35-925c-939e82694732-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41939f60-hbqs4\" (UID: \"9c92c3cd-9053-4a35-925c-939e82694732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"
Oct 31 13:36:00.060921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.060890    4428 generic.go:356] "Generic (PLEG): container finished" podID="9162cb66-9fcd-4fa4-89d2-f7044b8870a4" containerID="f9dabf4f694b7e704015f5086384a9a623c2a0c749c72a6ad58c181011ab947b" exitCode=0
Oct 31 13:36:00.061106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.060941    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml" event={"ID":"9162cb66-9fcd-4fa4-89d2-f7044b8870a4","Type":"ContainerDied","Data":"f9dabf4f694b7e704015f5086384a9a623c2a0c749c72a6ad58c181011ab947b"}
Oct 31 13:36:00.063066 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.063045    4428 generic.go:356] "Generic (PLEG): container finished" podID="8baa85c2-ada3-42ca-849c-afffc3e1c13e" containerID="976014cb385c6e29d29f15c4ef4ccc130e285dd7760e134bbe835b2b582e1ff4" exitCode=0
Oct 31 13:36:00.063207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.063121    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b" event={"ID":"8baa85c2-ada3-42ca-849c-afffc3e1c13e","Type":"ContainerDied","Data":"976014cb385c6e29d29f15c4ef4ccc130e285dd7760e134bbe835b2b582e1ff4"}
Oct 31 13:36:00.095980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.094399    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk67d\" (UniqueName: \"kubernetes.io/projected/9c92c3cd-9053-4a35-925c-939e82694732-kube-api-access-dk67d\") pod \"maas-api-revoke-sub-41939f60-hbqs4\" (UID: \"9c92c3cd-9053-4a35-925c-939e82694732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"
Oct 31 13:36:00.095980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.094727    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c92c3cd-9053-4a35-925c-939e82694732-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41939f60-hbqs4\" (UID: \"9c92c3cd-9053-4a35-925c-939e82694732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"
Oct 31 13:36:00.095980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.095601    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c92c3cd-9053-4a35-925c-939e82694732-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41939f60-hbqs4\" (UID: \"9c92c3cd-9053-4a35-925c-939e82694732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"
Oct 31 13:36:00.095980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.095702    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c92c3cd-9053-4a35-925c-939e82694732-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41939f60-hbqs4\" (UID: \"9c92c3cd-9053-4a35-925c-939e82694732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"
Oct 31 13:36:00.098633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.098607    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c92c3cd-9053-4a35-925c-939e82694732-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41939f60-hbqs4\" (UID: \"9c92c3cd-9053-4a35-925c-939e82694732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"
Oct 31 13:36:00.116560 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.116101    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk67d\" (UniqueName: \"kubernetes.io/projected/9c92c3cd-9053-4a35-925c-939e82694732-kube-api-access-dk67d\") pod \"maas-api-revoke-sub-41939f60-hbqs4\" (UID: \"9c92c3cd-9053-4a35-925c-939e82694732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"
Oct 31 13:36:00.209598 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.209302    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"
Oct 31 13:36:00.221346 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.221321    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"
Oct 31 13:36:00.305919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.301919    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9162cb66-9fcd-4fa4-89d2-f7044b8870a4-kube-api-access-sxlxf\" (UniqueName: \"kubernetes.io/projected/9162cb66-9fcd-4fa4-89d2-f7044b8870a4-kube-api-access-sxlxf\") pod \"9162cb66-9fcd-4fa4-89d2-f7044b8870a4\" (UID: \"9162cb66-9fcd-4fa4-89d2-f7044b8870a4\") "
Oct 31 13:36:00.305919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.302198    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8baa85c2-ada3-42ca-849c-afffc3e1c13e-kube-api-access-fwj9l\" (UniqueName: \"kubernetes.io/projected/8baa85c2-ada3-42ca-849c-afffc3e1c13e-kube-api-access-fwj9l\") pod \"8baa85c2-ada3-42ca-849c-afffc3e1c13e\" (UID: \"8baa85c2-ada3-42ca-849c-afffc3e1c13e\") "
Oct 31 13:36:00.305919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.302687    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8baa85c2-ada3-42ca-849c-afffc3e1c13e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8baa85c2-ada3-42ca-849c-afffc3e1c13e-plugins-config-volume\") pod \"8baa85c2-ada3-42ca-849c-afffc3e1c13e\" (UID: \"8baa85c2-ada3-42ca-849c-afffc3e1c13e\") "
Oct 31 13:36:00.305919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.302793    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9162cb66-9fcd-4fa4-89d2-f7044b8870a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9162cb66-9fcd-4fa4-89d2-f7044b8870a4-plugins-config-volume\") pod \"9162cb66-9fcd-4fa4-89d2-f7044b8870a4\" (UID: \"9162cb66-9fcd-4fa4-89d2-f7044b8870a4\") "
Oct 31 13:36:00.305919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.303221    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9162cb66-9fcd-4fa4-89d2-f7044b8870a4-plugins-config-volume" pod "9162cb66-9fcd-4fa4-89d2-f7044b8870a4" (UID: "9162cb66-9fcd-4fa4-89d2-f7044b8870a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:00.305919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.303220    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8baa85c2-ada3-42ca-849c-afffc3e1c13e-plugins-config-volume" pod "8baa85c2-ada3-42ca-849c-afffc3e1c13e" (UID: "8baa85c2-ada3-42ca-849c-afffc3e1c13e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:00.305919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.303701    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9162cb66-9fcd-4fa4-89d2-f7044b8870a4-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:00.305919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.303719    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8baa85c2-ada3-42ca-849c-afffc3e1c13e-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:00.307348 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.307313    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8baa85c2-ada3-42ca-849c-afffc3e1c13e-kube-api-access-fwj9l" pod "8baa85c2-ada3-42ca-849c-afffc3e1c13e" (UID: "8baa85c2-ada3-42ca-849c-afffc3e1c13e"). InnerVolumeSpecName "kube-api-access-fwj9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:00.309308 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.309276    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9162cb66-9fcd-4fa4-89d2-f7044b8870a4-kube-api-access-sxlxf" pod "9162cb66-9fcd-4fa4-89d2-f7044b8870a4" (UID: "9162cb66-9fcd-4fa4-89d2-f7044b8870a4"). InnerVolumeSpecName "kube-api-access-sxlxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:00.329005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.328968    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"
Oct 31 13:36:00.406166 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.406132    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxlxf\" (UniqueName: \"kubernetes.io/projected/9162cb66-9fcd-4fa4-89d2-f7044b8870a4-kube-api-access-sxlxf\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:00.406392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.406373    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwj9l\" (UniqueName: \"kubernetes.io/projected/8baa85c2-ada3-42ca-849c-afffc3e1c13e-kube-api-access-fwj9l\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:00.445568 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.445540    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:00.561015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.560938    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"]
Oct 31 13:36:00.565815 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.565790    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-79f4fb0f-g9wnh"]
Oct 31 13:36:00.615167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.615133    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:00.641348 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:36:00.641316    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c92c3cd_9053_4a35_925c_939e82694732.slice/crio-d273c2e84f178b59a92db024df273c67c480d42f8cc81135a8411e6c9ce14236 WatchSource:0}: Error finding container d273c2e84f178b59a92db024df273c67c480d42f8cc81135a8411e6c9ce14236: Status 404 returned error can't find the container with id d273c2e84f178b59a92db024df273c67c480d42f8cc81135a8411e6c9ce14236
Oct 31 13:36:00.644609 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.644579    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"]
Oct 31 13:36:00.845208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.842470    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:00.930905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:00.927544    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d136cbb-2350-444a-82c3-5b595212315f" path="/var/lib/kubelet/pods/4d136cbb-2350-444a-82c3-5b595212315f/volumes"
Oct 31 13:36:01.104696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:01.104589    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml" event={"ID":"9162cb66-9fcd-4fa4-89d2-f7044b8870a4","Type":"ContainerDied","Data":"397fc871cea58717cad81f0558f3ea3aedb30656e2d8e7427830f3526b0f4680"}
Oct 31 13:36:01.104998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:01.104973    4428 scope.go:122] "RemoveContainer" containerID="f9dabf4f694b7e704015f5086384a9a623c2a0c749c72a6ad58c181011ab947b"
Oct 31 13:36:01.105306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:01.105288    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"
Oct 31 13:36:01.135559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:01.135501    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:01.193892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:01.173704    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4" event={"ID":"9c92c3cd-9053-4a35-925c-939e82694732","Type":"ContainerStarted","Data":"d273c2e84f178b59a92db024df273c67c480d42f8cc81135a8411e6c9ce14236"}
Oct 31 13:36:01.195755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:01.194724    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b" event={"ID":"8baa85c2-ada3-42ca-849c-afffc3e1c13e","Type":"ContainerDied","Data":"7656b23a0dfee736c3ec4bfc930874ab28568588d344186df8ac4c71e5877d49"}
Oct 31 13:36:01.195755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:01.194744    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"
Oct 31 13:36:01.220782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:01.217943    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"]
Oct 31 13:36:01.226595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:01.223851    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-752869-67c9855ff-xz5ml"]
Oct 31 13:36:01.234174 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:01.231010    4428 scope.go:122] "RemoveContainer" containerID="976014cb385c6e29d29f15c4ef4ccc130e285dd7760e134bbe835b2b582e1ff4"
Oct 31 13:36:01.245418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:01.245368    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"]
Oct 31 13:36:01.251744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:01.251665    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-752869-7c98d67ff-qnz9b"]
Oct 31 13:36:01.293659 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:01.293630    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"]
Oct 31 13:36:01.298276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:01.298249    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-75c956-4vj88"]
Oct 31 13:36:01.337222 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:01.337191    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:01.524434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:01.517958    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:01.869078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:01.869052    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:02.205620 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:02.205524    4428 generic.go:356] "Generic (PLEG): container finished" podID="9c92c3cd-9053-4a35-925c-939e82694732" containerID="00793ed6d6235d8a605668c66fc76e94b1e39d6cd20c75a6ed1c7dc860aed721" exitCode=0
Oct 31 13:36:02.205889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:02.205618    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4" event={"ID":"9c92c3cd-9053-4a35-925c-939e82694732","Type":"ContainerDied","Data":"00793ed6d6235d8a605668c66fc76e94b1e39d6cd20c75a6ed1c7dc860aed721"}
Oct 31 13:36:02.498749 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:02.473937    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:02.760121 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:02.760030    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080d9f1e-4bc4-4413-9cf9-80e1b7ac941e" path="/var/lib/kubelet/pods/080d9f1e-4bc4-4413-9cf9-80e1b7ac941e/volumes"
Oct 31 13:36:02.760706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:02.760678    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8baa85c2-ada3-42ca-849c-afffc3e1c13e" path="/var/lib/kubelet/pods/8baa85c2-ada3-42ca-849c-afffc3e1c13e/volumes"
Oct 31 13:36:02.761249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:02.761228    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9162cb66-9fcd-4fa4-89d2-f7044b8870a4" path="/var/lib/kubelet/pods/9162cb66-9fcd-4fa4-89d2-f7044b8870a4/volumes"
Oct 31 13:36:03.405890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:03.397652    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:03.686138 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:03.686092    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88" podUID="6cda6b31-2ad2-4248-aa7b-7ca7a25ef681" containerName="payload-pre-processing" containerID="cri-o://3bac2c5ff9fdff976cff5d8a5fcfacbc213980682c8f05f82d9389cd9a1b22ba" gracePeriod=25
Oct 31 13:36:03.828312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:03.828272    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"
Oct 31 13:36:03.856389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:03.856007    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c92c3cd-9053-4a35-925c-939e82694732-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c92c3cd-9053-4a35-925c-939e82694732-maas-api-xks-ca\") pod \"9c92c3cd-9053-4a35-925c-939e82694732\" (UID: \"9c92c3cd-9053-4a35-925c-939e82694732\") "
Oct 31 13:36:03.856389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:03.856095    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c92c3cd-9053-4a35-925c-939e82694732-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c92c3cd-9053-4a35-925c-939e82694732-maas-api-openshift-ca\") pod \"9c92c3cd-9053-4a35-925c-939e82694732\" (UID: \"9c92c3cd-9053-4a35-925c-939e82694732\") "
Oct 31 13:36:03.856389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:03.856188    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c92c3cd-9053-4a35-925c-939e82694732-kube-api-access-dk67d\" (UniqueName: \"kubernetes.io/projected/9c92c3cd-9053-4a35-925c-939e82694732-kube-api-access-dk67d\") pod \"9c92c3cd-9053-4a35-925c-939e82694732\" (UID: \"9c92c3cd-9053-4a35-925c-939e82694732\") "
Oct 31 13:36:03.858696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:03.858451    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c92c3cd-9053-4a35-925c-939e82694732-maas-api-openshift-ca" pod "9c92c3cd-9053-4a35-925c-939e82694732" (UID: "9c92c3cd-9053-4a35-925c-939e82694732"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:03.860978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:03.860858    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c92c3cd-9053-4a35-925c-939e82694732-maas-api-xks-ca" pod "9c92c3cd-9053-4a35-925c-939e82694732" (UID: "9c92c3cd-9053-4a35-925c-939e82694732"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:36:03.860978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:03.860945    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c92c3cd-9053-4a35-925c-939e82694732-kube-api-access-dk67d" pod "9c92c3cd-9053-4a35-925c-939e82694732" (UID: "9c92c3cd-9053-4a35-925c-939e82694732"). InnerVolumeSpecName "kube-api-access-dk67d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:03.958923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:03.958892    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c92c3cd-9053-4a35-925c-939e82694732-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:03.958923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:03.958921    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c92c3cd-9053-4a35-925c-939e82694732-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:03.959164 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:03.958932    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk67d\" (UniqueName: \"kubernetes.io/projected/9c92c3cd-9053-4a35-925c-939e82694732-kube-api-access-dk67d\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:04.049986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.049955    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"
Oct 31 13:36:04.161631 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.161550    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cda6b31-2ad2-4248-aa7b-7ca7a25ef681-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cda6b31-2ad2-4248-aa7b-7ca7a25ef681-plugins-config-volume\") pod \"6cda6b31-2ad2-4248-aa7b-7ca7a25ef681\" (UID: \"6cda6b31-2ad2-4248-aa7b-7ca7a25ef681\") "
Oct 31 13:36:04.161818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.161716    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cda6b31-2ad2-4248-aa7b-7ca7a25ef681-kube-api-access-7sxjw\" (UniqueName: \"kubernetes.io/projected/6cda6b31-2ad2-4248-aa7b-7ca7a25ef681-kube-api-access-7sxjw\") pod \"6cda6b31-2ad2-4248-aa7b-7ca7a25ef681\" (UID: \"6cda6b31-2ad2-4248-aa7b-7ca7a25ef681\") "
Oct 31 13:36:04.161992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.161965    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cda6b31-2ad2-4248-aa7b-7ca7a25ef681-plugins-config-volume" pod "6cda6b31-2ad2-4248-aa7b-7ca7a25ef681" (UID: "6cda6b31-2ad2-4248-aa7b-7ca7a25ef681"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:04.162291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.162273    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cda6b31-2ad2-4248-aa7b-7ca7a25ef681-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:04.163996 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.163973    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cda6b31-2ad2-4248-aa7b-7ca7a25ef681-kube-api-access-7sxjw" pod "6cda6b31-2ad2-4248-aa7b-7ca7a25ef681" (UID: "6cda6b31-2ad2-4248-aa7b-7ca7a25ef681"). InnerVolumeSpecName "kube-api-access-7sxjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:04.248421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.248388    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"
Oct 31 13:36:04.248584 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.248421    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4" event={"ID":"9c92c3cd-9053-4a35-925c-939e82694732","Type":"ContainerDied","Data":"d273c2e84f178b59a92db024df273c67c480d42f8cc81135a8411e6c9ce14236"}
Oct 31 13:36:04.248584 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.248472    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d273c2e84f178b59a92db024df273c67c480d42f8cc81135a8411e6c9ce14236"
Oct 31 13:36:04.249921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.249893    4428 generic.go:356] "Generic (PLEG): container finished" podID="6cda6b31-2ad2-4248-aa7b-7ca7a25ef681" containerID="3bac2c5ff9fdff976cff5d8a5fcfacbc213980682c8f05f82d9389cd9a1b22ba" exitCode=0
Oct 31 13:36:04.250040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.249929    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"
Oct 31 13:36:04.250040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.249966    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88" event={"ID":"6cda6b31-2ad2-4248-aa7b-7ca7a25ef681","Type":"ContainerDied","Data":"3bac2c5ff9fdff976cff5d8a5fcfacbc213980682c8f05f82d9389cd9a1b22ba"}
Oct 31 13:36:04.250040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.250019    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88" event={"ID":"6cda6b31-2ad2-4248-aa7b-7ca7a25ef681","Type":"ContainerDied","Data":"d0af413a64add90ce8ea29ab2474e3edd03d15db8182e2bd932afa8287c1785b"}
Oct 31 13:36:04.250170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.250046    4428 scope.go:122] "RemoveContainer" containerID="3bac2c5ff9fdff976cff5d8a5fcfacbc213980682c8f05f82d9389cd9a1b22ba"
Oct 31 13:36:04.264190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.264163    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sxjw\" (UniqueName: \"kubernetes.io/projected/6cda6b31-2ad2-4248-aa7b-7ca7a25ef681-kube-api-access-7sxjw\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:04.269665 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.269584    4428 scope.go:122] "RemoveContainer" containerID="3bac2c5ff9fdff976cff5d8a5fcfacbc213980682c8f05f82d9389cd9a1b22ba"
Oct 31 13:36:04.270292 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:04.270263    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bac2c5ff9fdff976cff5d8a5fcfacbc213980682c8f05f82d9389cd9a1b22ba\": container with ID starting with 3bac2c5ff9fdff976cff5d8a5fcfacbc213980682c8f05f82d9389cd9a1b22ba not found: ID does not exist" containerID="3bac2c5ff9fdff976cff5d8a5fcfacbc213980682c8f05f82d9389cd9a1b22ba"
Oct 31 13:36:04.270396 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.270302    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bac2c5ff9fdff976cff5d8a5fcfacbc213980682c8f05f82d9389cd9a1b22ba"} err="failed to get container status \"3bac2c5ff9fdff976cff5d8a5fcfacbc213980682c8f05f82d9389cd9a1b22ba\": rpc error: code = NotFound desc = could not find container \"3bac2c5ff9fdff976cff5d8a5fcfacbc213980682c8f05f82d9389cd9a1b22ba\": container with ID starting with 3bac2c5ff9fdff976cff5d8a5fcfacbc213980682c8f05f82d9389cd9a1b22ba not found: ID does not exist"
Oct 31 13:36:04.284680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.284639    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"]
Oct 31 13:36:04.290094 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.290070    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-fnt88"]
Oct 31 13:36:04.612892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.607762    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:04.767216 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:04.767185    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cda6b31-2ad2-4248-aa7b-7ca7a25ef681" path="/var/lib/kubelet/pods/6cda6b31-2ad2-4248-aa7b-7ca7a25ef681/volumes"
Oct 31 13:36:05.085821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:05.085474    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:05.467887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:05.464718    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:05.747179 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:05.747103    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:06.268500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:06.268455    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:06.438899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:06.438849    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:07.733486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:07.733428    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:08.648908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:08.642890    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:09.697831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:09.697800    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:09.880406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:09.880325    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:10.230117 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:10.230076    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice/crio-91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15\": RecentStats: unable to find data in memory cache]"
Oct 31 13:36:10.419901 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:10.417691    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice/crio-91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15\": RecentStats: unable to find data in memory cache]"
Oct 31 13:36:10.423527 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:10.423493    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:10.587569 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:10.587472    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:10.776634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:10.776594    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:10.967325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:10.967293    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:11.250014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.249929    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"]
Oct 31 13:36:11.251663 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.251610    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9162cb66-9fcd-4fa4-89d2-f7044b8870a4" containerName="payload-pre-processing"
Oct 31 13:36:11.255944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.255917    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9162cb66-9fcd-4fa4-89d2-f7044b8870a4" containerName="payload-pre-processing"
Oct 31 13:36:11.256183 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.256167    4428 cpu_manager.go:414] "containerMap: removing container" podUID="8baa85c2-ada3-42ca-849c-afffc3e1c13e" containerName="payload-processing"
Oct 31 13:36:11.256279 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.256268    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8baa85c2-ada3-42ca-849c-afffc3e1c13e" containerName="payload-processing"
Oct 31 13:36:11.256431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.256422    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9c92c3cd-9053-4a35-925c-939e82694732" containerName="revoke-keys"
Oct 31 13:36:11.256507 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.256497    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c92c3cd-9053-4a35-925c-939e82694732" containerName="revoke-keys"
Oct 31 13:36:11.256585 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.256574    4428 cpu_manager.go:414] "containerMap: removing container" podUID="6cda6b31-2ad2-4248-aa7b-7ca7a25ef681" containerName="payload-pre-processing"
Oct 31 13:36:11.256671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.256661    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cda6b31-2ad2-4248-aa7b-7ca7a25ef681" containerName="payload-pre-processing"
Oct 31 13:36:11.257008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.256992    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9162cb66-9fcd-4fa4-89d2-f7044b8870a4" containerName="payload-pre-processing"
Oct 31 13:36:11.257137 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.257124    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8baa85c2-ada3-42ca-849c-afffc3e1c13e" containerName="payload-processing"
Oct 31 13:36:11.257247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.257237    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c92c3cd-9053-4a35-925c-939e82694732" containerName="revoke-keys"
Oct 31 13:36:11.257328 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.257318    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cda6b31-2ad2-4248-aa7b-7ca7a25ef681" containerName="payload-pre-processing"
Oct 31 13:36:11.430149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.430100    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:11.430427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.430133    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"]
Oct 31 13:36:11.430570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.430478    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"
Oct 31 13:36:11.438512 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.438491    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:36:11.439027 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.439009    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:36:11.482856 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.482828    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"]
Oct 31 13:36:11.488760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.488653    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"
Oct 31 13:36:11.495861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.495836    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c45a37-dockercfg-llxkp\""
Oct 31 13:36:11.497613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.497588    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"]
Oct 31 13:36:11.532467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.532389    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82cj9\" (UniqueName: \"kubernetes.io/projected/c934f1a7-3712-4955-ad79-5505b3760938-kube-api-access-82cj9\") pod \"maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7\" (UID: \"c934f1a7-3712-4955-ad79-5505b3760938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"
Oct 31 13:36:11.532616 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.532572    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpjpq\" (UniqueName: \"kubernetes.io/projected/8a7e177f-205f-4ba6-8711-40a0b52158ea-kube-api-access-mpjpq\") pod \"payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj\" (UID: \"8a7e177f-205f-4ba6-8711-40a0b52158ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"
Oct 31 13:36:11.532720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.532699    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c934f1a7-3712-4955-ad79-5505b3760938-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7\" (UID: \"c934f1a7-3712-4955-ad79-5505b3760938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"
Oct 31 13:36:11.532813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.532787    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7e177f-205f-4ba6-8711-40a0b52158ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj\" (UID: \"8a7e177f-205f-4ba6-8711-40a0b52158ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"
Oct 31 13:36:11.532897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.532884    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c934f1a7-3712-4955-ad79-5505b3760938-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7\" (UID: \"c934f1a7-3712-4955-ad79-5505b3760938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"
Oct 31 13:36:11.537853 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.537829    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c45a37\""
Oct 31 13:36:11.607928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.605076    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:11.674942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.665267    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82cj9\" (UniqueName: \"kubernetes.io/projected/c934f1a7-3712-4955-ad79-5505b3760938-kube-api-access-82cj9\") pod \"maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7\" (UID: \"c934f1a7-3712-4955-ad79-5505b3760938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"
Oct 31 13:36:11.674942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.665433    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpjpq\" (UniqueName: \"kubernetes.io/projected/8a7e177f-205f-4ba6-8711-40a0b52158ea-kube-api-access-mpjpq\") pod \"payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj\" (UID: \"8a7e177f-205f-4ba6-8711-40a0b52158ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"
Oct 31 13:36:11.674942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.665549    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c934f1a7-3712-4955-ad79-5505b3760938-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7\" (UID: \"c934f1a7-3712-4955-ad79-5505b3760938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"
Oct 31 13:36:11.674942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.665594    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7e177f-205f-4ba6-8711-40a0b52158ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj\" (UID: \"8a7e177f-205f-4ba6-8711-40a0b52158ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"
Oct 31 13:36:11.674942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.665649    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c934f1a7-3712-4955-ad79-5505b3760938-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7\" (UID: \"c934f1a7-3712-4955-ad79-5505b3760938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"
Oct 31 13:36:11.733958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.732969    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c934f1a7-3712-4955-ad79-5505b3760938-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7\" (UID: \"c934f1a7-3712-4955-ad79-5505b3760938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"
Oct 31 13:36:11.733958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.733690    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7e177f-205f-4ba6-8711-40a0b52158ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj\" (UID: \"8a7e177f-205f-4ba6-8711-40a0b52158ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"
Oct 31 13:36:11.763914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.735342    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c934f1a7-3712-4955-ad79-5505b3760938-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7\" (UID: \"c934f1a7-3712-4955-ad79-5505b3760938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"
Oct 31 13:36:11.827978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.827554    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpjpq\" (UniqueName: \"kubernetes.io/projected/8a7e177f-205f-4ba6-8711-40a0b52158ea-kube-api-access-mpjpq\") pod \"payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj\" (UID: \"8a7e177f-205f-4ba6-8711-40a0b52158ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"
Oct 31 13:36:11.828489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:11.828195    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82cj9\" (UniqueName: \"kubernetes.io/projected/c934f1a7-3712-4955-ad79-5505b3760938-kube-api-access-82cj9\") pod \"maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7\" (UID: \"c934f1a7-3712-4955-ad79-5505b3760938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"
Oct 31 13:36:12.059902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:12.059192    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"
Oct 31 13:36:12.109400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:12.109364    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"
Oct 31 13:36:12.356078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:12.354423    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:12.452830 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:36:12.452760    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc934f1a7_3712_4955_ad79_5505b3760938.slice/crio-e2aa29b2ae5f919b85008b06264475b571030e23e452d73a606763e938bb9218 WatchSource:0}: Error finding container e2aa29b2ae5f919b85008b06264475b571030e23e452d73a606763e938bb9218: Status 404 returned error can't find the container with id e2aa29b2ae5f919b85008b06264475b571030e23e452d73a606763e938bb9218
Oct 31 13:36:12.467655 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:12.467601    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"]
Oct 31 13:36:12.959305 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:12.959267    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"]
Oct 31 13:36:13.208666 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:13.208635    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:13.480972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:13.480933    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj" event={"ID":"8a7e177f-205f-4ba6-8711-40a0b52158ea","Type":"ContainerStarted","Data":"4902bf17e7985d36132b6f596a5436dc6901b48efd6a339eeeffba2e73941c29"}
Oct 31 13:36:13.481141 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:13.480995    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj" event={"ID":"8a7e177f-205f-4ba6-8711-40a0b52158ea","Type":"ContainerStarted","Data":"5b646827e64bf81ea0d960bda71727f80567c32be8c9e854870770b00109585e"}
Oct 31 13:36:13.481229 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:13.481209    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"
Oct 31 13:36:13.482671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:13.482640    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7" event={"ID":"c934f1a7-3712-4955-ad79-5505b3760938","Type":"ContainerStarted","Data":"293da6842dade8d063f319ae06d6be691257d1d8f52e89184d3070256363797c"}
Oct 31 13:36:13.482802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:13.482692    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7" event={"ID":"c934f1a7-3712-4955-ad79-5505b3760938","Type":"ContainerStarted","Data":"e2aa29b2ae5f919b85008b06264475b571030e23e452d73a606763e938bb9218"}
Oct 31 13:36:13.574892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:13.574241    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj" podStartSLOduration=2.574223048 podStartE2EDuration="2.574223048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:13.537797644 +0000 UTC m=+1970.936146268" watchObservedRunningTime="2026-10-31 13:36:13.574223048 +0000 UTC m=+1970.972571672"
Oct 31 13:36:13.605892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:13.605028    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7" podStartSLOduration=2.605009827 podStartE2EDuration="2.605009827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:13.563717185 +0000 UTC m=+1970.962065821" watchObservedRunningTime="2026-10-31 13:36:13.605009827 +0000 UTC m=+1971.003358455"
Oct 31 13:36:13.991797 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:13.991707    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:14.520896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:14.517811    4428 generic.go:356] "Generic (PLEG): container finished" podID="c934f1a7-3712-4955-ad79-5505b3760938" containerID="293da6842dade8d063f319ae06d6be691257d1d8f52e89184d3070256363797c" exitCode=0
Oct 31 13:36:14.520896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:14.519473    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7" event={"ID":"c934f1a7-3712-4955-ad79-5505b3760938","Type":"ContainerDied","Data":"293da6842dade8d063f319ae06d6be691257d1d8f52e89184d3070256363797c"}
Oct 31 13:36:14.710904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:14.710532    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"]
Oct 31 13:36:14.711550 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:14.711094    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn" podUID="69e04722-598c-4608-9742-440e327e389a" containerName="istio-proxy" containerID="cri-o://3a3470b0c35f73c4ef5c10cbe6cadf696674bebc1230b9fe1d147f3791082c1f" gracePeriod=30
Oct 31 13:36:14.723592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:14.723559    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"]
Oct 31 13:36:14.723823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:14.723799    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" podUID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" containerName="istio-proxy" containerID="cri-o://3c0f0842e7377dd501d6d955f7aab95cbba47bf1f6a3ea579bad79d33b4b773a" gracePeriod=30
Oct 31 13:36:14.728721 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:14.728652    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"]
Oct 31 13:36:14.729086 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:14.729060    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns" podUID="93314798-1f48-408a-b87c-d8f809d2f14b" containerName="istio-proxy" containerID="cri-o://efa22e7ed2f6b7bf67c47aba76e075d62775b017722de4af5944aa9e1545e58f" gracePeriod=30
Oct 31 13:36:14.733738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:14.733708    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"]
Oct 31 13:36:14.733961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:14.733936    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" containerID="cri-o://8f11954f00bcb576c130524e86f1593ea416ce35774aff28da5f3613f54b13f7" gracePeriod=30
Oct 31 13:36:14.738085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:14.738061    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"]
Oct 31 13:36:14.738261 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:14.738241    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2" podUID="9796198f-74dc-491b-b5fe-7ee7bb9ae809" containerName="istio-proxy" containerID="cri-o://d171285953525cf3a208e44c8fba973804fa5563b3413e5785d39cb6f371afa7" gracePeriod=30
Oct 31 13:36:14.742786 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:14.742764    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"]
Oct 31 13:36:14.742945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:14.742924    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" containerID="cri-o://f42e6e218cf025c36b4d6b0ebf897e58ccfe565b42aef9fe335c7cbb8fd7d71a" gracePeriod=30
Oct 31 13:36:15.010745 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:15.010710    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"]
Oct 31 13:36:15.011269 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:15.011076    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy" containerID="cri-o://663b7fae31d3997e9a3aa2e3842bac417d1b7aa1223a719ba0603aa3645d1645" gracePeriod=30
Oct 31 13:36:15.019042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:15.019015    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"]
Oct 31 13:36:15.019482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:15.019455    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt" podUID="a64a3776-6565-48d0-8c36-5c4d979eeb02" containerName="istio-proxy" containerID="cri-o://20d158b99f4b4906ebae7ac5a24bf7263fade6f80cffe811f49432955e146da3" gracePeriod=30
Oct 31 13:36:15.028973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:15.028943    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"]
Oct 31 13:36:15.029174 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:15.029151    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx" podUID="aaab76aa-bfe5-4413-83a8-bc1037c2b525" containerName="istio-proxy" containerID="cri-o://bdee988fff66234b73085f7d21cb3c50213f9a09b6e07328bb1720a57d4f3fab" gracePeriod=30
Oct 31 13:36:15.833991 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:15.826399    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:16.121842 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:16.120011    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"
Oct 31 13:36:16.167896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:16.165395    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c934f1a7-3712-4955-ad79-5505b3760938-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c934f1a7-3712-4955-ad79-5505b3760938-maas-api-xks-ca\") pod \"c934f1a7-3712-4955-ad79-5505b3760938\" (UID: \"c934f1a7-3712-4955-ad79-5505b3760938\") "
Oct 31 13:36:16.167896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:16.165551    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c934f1a7-3712-4955-ad79-5505b3760938-kube-api-access-82cj9\" (UniqueName: \"kubernetes.io/projected/c934f1a7-3712-4955-ad79-5505b3760938-kube-api-access-82cj9\") pod \"c934f1a7-3712-4955-ad79-5505b3760938\" (UID: \"c934f1a7-3712-4955-ad79-5505b3760938\") "
Oct 31 13:36:16.167896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:16.165762    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c934f1a7-3712-4955-ad79-5505b3760938-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c934f1a7-3712-4955-ad79-5505b3760938-maas-api-openshift-ca\") pod \"c934f1a7-3712-4955-ad79-5505b3760938\" (UID: \"c934f1a7-3712-4955-ad79-5505b3760938\") "
Oct 31 13:36:16.189714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:16.169359    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c934f1a7-3712-4955-ad79-5505b3760938-maas-api-openshift-ca" pod "c934f1a7-3712-4955-ad79-5505b3760938" (UID: "c934f1a7-3712-4955-ad79-5505b3760938"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:16.189714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:16.174932    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c934f1a7-3712-4955-ad79-5505b3760938-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:16.209894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:16.200393    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c934f1a7-3712-4955-ad79-5505b3760938-maas-api-xks-ca" pod "c934f1a7-3712-4955-ad79-5505b3760938" (UID: "c934f1a7-3712-4955-ad79-5505b3760938"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:36:16.213160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:16.213088    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c934f1a7-3712-4955-ad79-5505b3760938-kube-api-access-82cj9" pod "c934f1a7-3712-4955-ad79-5505b3760938" (UID: "c934f1a7-3712-4955-ad79-5505b3760938"). InnerVolumeSpecName "kube-api-access-82cj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:16.301951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:16.280001    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c934f1a7-3712-4955-ad79-5505b3760938-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:16.301951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:16.280047    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82cj9\" (UniqueName: \"kubernetes.io/projected/c934f1a7-3712-4955-ad79-5505b3760938-kube-api-access-82cj9\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:16.538580 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:16.538544    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7" event={"ID":"c934f1a7-3712-4955-ad79-5505b3760938","Type":"ContainerDied","Data":"e2aa29b2ae5f919b85008b06264475b571030e23e452d73a606763e938bb9218"}
Oct 31 13:36:16.538764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:16.538597    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2aa29b2ae5f919b85008b06264475b571030e23e452d73a606763e938bb9218"
Oct 31 13:36:16.538764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:16.538728    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"
Oct 31 13:36:18.118894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:18.114761    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:18.741818 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:18.741764    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:36:18.775892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:18.769212    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:19.094948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:19.094386    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:36:19.094948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:19.094457    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:36:19.110554 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:19.110527    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:36:19.110730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:19.110705    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:36:19.534882 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:19.534697    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:19.943466 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:19.943409    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e04722_598c_4608_9742_440e327e389a.slice/crio-conmon-3a3470b0c35f73c4ef5c10cbe6cadf696674bebc1230b9fe1d147f3791082c1f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8836a12_5c77_42e2_bb58_e69be67808db.slice/crio-8f11954f00bcb576c130524e86f1593ea416ce35774aff28da5f3613f54b13f7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93314798_1f48_408a_b87c_d8f809d2f14b.slice/crio-conmon-efa22e7ed2f6b7bf67c47aba76e075d62775b017722de4af5944aa9e1545e58f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93314798_1f48_408a_b87c_d8f809d2f14b.slice/crio-efa22e7ed2f6b7bf67c47aba76e075d62775b017722de4af5944aa9e1545e58f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice/crio-91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15\": RecentStats: unable to find data in memory cache]"
Oct 31 13:36:19.943584 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:19.943475    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e04722_598c_4608_9742_440e327e389a.slice/crio-conmon-3a3470b0c35f73c4ef5c10cbe6cadf696674bebc1230b9fe1d147f3791082c1f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93314798_1f48_408a_b87c_d8f809d2f14b.slice/crio-conmon-efa22e7ed2f6b7bf67c47aba76e075d62775b017722de4af5944aa9e1545e58f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8836a12_5c77_42e2_bb58_e69be67808db.slice/crio-8f11954f00bcb576c130524e86f1593ea416ce35774aff28da5f3613f54b13f7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice/crio-91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4071546e_73df_4d16_a000_36fc2372f54b.slice/crio-conmon-f42e6e218cf025c36b4d6b0ebf897e58ccfe565b42aef9fe335c7cbb8fd7d71a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e04722_598c_4608_9742_440e327e389a.slice/crio-3a3470b0c35f73c4ef5c10cbe6cadf696674bebc1230b9fe1d147f3791082c1f.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 13:36:20.338051 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:20.321321    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice/crio-91c04fd3f860943444cc5f3325efad00ebc5f6a4ea7996ea5b95ac57f6405f15\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d9f1e_4bc4_4413_9cf9_80e1b7ac941e.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:36:20.864006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.863520    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:36:20.939353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.939317    4428 generic.go:356] "Generic (PLEG): container finished" podID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" containerID="3c0f0842e7377dd501d6d955f7aab95cbba47bf1f6a3ea579bad79d33b4b773a" exitCode=0
Oct 31 13:36:20.939476 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.939428    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" event={"ID":"af5e815e-eb38-40c4-9651-e4dbbb92dc36","Type":"ContainerDied","Data":"3c0f0842e7377dd501d6d955f7aab95cbba47bf1f6a3ea579bad79d33b4b773a"}
Oct 31 13:36:20.941811 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.941627    4428 generic.go:356] "Generic (PLEG): container finished" podID="9796198f-74dc-491b-b5fe-7ee7bb9ae809" containerID="d171285953525cf3a208e44c8fba973804fa5563b3413e5785d39cb6f371afa7" exitCode=0
Oct 31 13:36:20.941811 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.941745    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2" event={"ID":"9796198f-74dc-491b-b5fe-7ee7bb9ae809","Type":"ContainerDied","Data":"d171285953525cf3a208e44c8fba973804fa5563b3413e5785d39cb6f371afa7"}
Oct 31 13:36:20.944017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.943813    4428 generic.go:356] "Generic (PLEG): container finished" podID="b8571392-d978-4512-939b-69b9fadb6734" containerID="663b7fae31d3997e9a3aa2e3842bac417d1b7aa1223a719ba0603aa3645d1645" exitCode=0
Oct 31 13:36:20.944017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.943955    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" event={"ID":"b8571392-d978-4512-939b-69b9fadb6734","Type":"ContainerDied","Data":"663b7fae31d3997e9a3aa2e3842bac417d1b7aa1223a719ba0603aa3645d1645"}
Oct 31 13:36:20.969892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.965067    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4071546e-73df-4d16-a000-36fc2372f54b-istio-token\" (UniqueName: \"kubernetes.io/projected/4071546e-73df-4d16-a000-36fc2372f54b-istio-token\") pod \"4071546e-73df-4d16-a000-36fc2372f54b\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") "
Oct 31 13:36:20.969892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.965122    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4071546e-73df-4d16-a000-36fc2372f54b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4071546e-73df-4d16-a000-36fc2372f54b-istio-podinfo\") pod \"4071546e-73df-4d16-a000-36fc2372f54b\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") "
Oct 31 13:36:20.969892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.965184    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-credential-socket\") pod \"4071546e-73df-4d16-a000-36fc2372f54b\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") "
Oct 31 13:36:20.969892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.965334    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-istio-envoy\") pod \"4071546e-73df-4d16-a000-36fc2372f54b\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") "
Oct 31 13:36:20.969892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.965375    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-workload-certs\") pod \"4071546e-73df-4d16-a000-36fc2372f54b\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") "
Oct 31 13:36:20.969892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.965430    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-istio-data\") pod \"4071546e-73df-4d16-a000-36fc2372f54b\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") "
Oct 31 13:36:20.969892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.965517    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4071546e-73df-4d16-a000-36fc2372f54b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4071546e-73df-4d16-a000-36fc2372f54b-istiod-ca-cert\") pod \"4071546e-73df-4d16-a000-36fc2372f54b\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") "
Oct 31 13:36:20.969892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.965581    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4071546e-73df-4d16-a000-36fc2372f54b-kube-api-access-bgclw\" (UniqueName: \"kubernetes.io/projected/4071546e-73df-4d16-a000-36fc2372f54b-kube-api-access-bgclw\") pod \"4071546e-73df-4d16-a000-36fc2372f54b\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") "
Oct 31 13:36:20.969892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.965611    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-workload-socket\") pod \"4071546e-73df-4d16-a000-36fc2372f54b\" (UID: \"4071546e-73df-4d16-a000-36fc2372f54b\") "
Oct 31 13:36:20.969892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.968473    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-workload-socket" pod "4071546e-73df-4d16-a000-36fc2372f54b" (UID: "4071546e-73df-4d16-a000-36fc2372f54b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:20.969892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.968718    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-istio-data" pod "4071546e-73df-4d16-a000-36fc2372f54b" (UID: "4071546e-73df-4d16-a000-36fc2372f54b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:20.969892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.969063    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4071546e-73df-4d16-a000-36fc2372f54b-istiod-ca-cert" pod "4071546e-73df-4d16-a000-36fc2372f54b" (UID: "4071546e-73df-4d16-a000-36fc2372f54b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:20.969892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.969632    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-workload-certs" pod "4071546e-73df-4d16-a000-36fc2372f54b" (UID: "4071546e-73df-4d16-a000-36fc2372f54b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:20.972198 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.971640    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-credential-socket" pod "4071546e-73df-4d16-a000-36fc2372f54b" (UID: "4071546e-73df-4d16-a000-36fc2372f54b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:20.973479 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.973196    4428 generic.go:356] "Generic (PLEG): container finished" podID="93314798-1f48-408a-b87c-d8f809d2f14b" containerID="efa22e7ed2f6b7bf67c47aba76e075d62775b017722de4af5944aa9e1545e58f" exitCode=0
Oct 31 13:36:20.973479 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.973397    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns" event={"ID":"93314798-1f48-408a-b87c-d8f809d2f14b","Type":"ContainerDied","Data":"efa22e7ed2f6b7bf67c47aba76e075d62775b017722de4af5944aa9e1545e58f"}
Oct 31 13:36:20.976107 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.975953    4428 generic.go:356] "Generic (PLEG): container finished" podID="69e04722-598c-4608-9742-440e327e389a" containerID="3a3470b0c35f73c4ef5c10cbe6cadf696674bebc1230b9fe1d147f3791082c1f" exitCode=0
Oct 31 13:36:20.976107 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.976049    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn" event={"ID":"69e04722-598c-4608-9742-440e327e389a","Type":"ContainerDied","Data":"3a3470b0c35f73c4ef5c10cbe6cadf696674bebc1230b9fe1d147f3791082c1f"}
Oct 31 13:36:20.977901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.977680    4428 generic.go:356] "Generic (PLEG): container finished" podID="4071546e-73df-4d16-a000-36fc2372f54b" containerID="f42e6e218cf025c36b4d6b0ebf897e58ccfe565b42aef9fe335c7cbb8fd7d71a" exitCode=0
Oct 31 13:36:20.977901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.977820    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" event={"ID":"4071546e-73df-4d16-a000-36fc2372f54b","Type":"ContainerDied","Data":"f42e6e218cf025c36b4d6b0ebf897e58ccfe565b42aef9fe335c7cbb8fd7d71a"}
Oct 31 13:36:20.978520 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.977859    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58" event={"ID":"4071546e-73df-4d16-a000-36fc2372f54b","Type":"ContainerDied","Data":"3df56d956594fb1ca0e8f45a82579e7cb758f5006c8989b32b71ec2f652a3c79"}
Oct 31 13:36:20.978520 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.978129    4428 scope.go:122] "RemoveContainer" containerID="f42e6e218cf025c36b4d6b0ebf897e58ccfe565b42aef9fe335c7cbb8fd7d71a"
Oct 31 13:36:20.978520 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.978308    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"
Oct 31 13:36:20.983275 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.982780    4428 generic.go:356] "Generic (PLEG): container finished" podID="aaab76aa-bfe5-4413-83a8-bc1037c2b525" containerID="bdee988fff66234b73085f7d21cb3c50213f9a09b6e07328bb1720a57d4f3fab" exitCode=0
Oct 31 13:36:20.983275 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.982908    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx" event={"ID":"aaab76aa-bfe5-4413-83a8-bc1037c2b525","Type":"ContainerDied","Data":"bdee988fff66234b73085f7d21cb3c50213f9a09b6e07328bb1720a57d4f3fab"}
Oct 31 13:36:20.986415 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.986273    4428 generic.go:356] "Generic (PLEG): container finished" podID="a8836a12-5c77-42e2-bb58-e69be67808db" containerID="8f11954f00bcb576c130524e86f1593ea416ce35774aff28da5f3613f54b13f7" exitCode=0
Oct 31 13:36:20.986415 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.986367    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" event={"ID":"a8836a12-5c77-42e2-bb58-e69be67808db","Type":"ContainerDied","Data":"8f11954f00bcb576c130524e86f1593ea416ce35774aff28da5f3613f54b13f7"}
Oct 31 13:36:20.988895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.988857    4428 generic.go:356] "Generic (PLEG): container finished" podID="a64a3776-6565-48d0-8c36-5c4d979eeb02" containerID="20d158b99f4b4906ebae7ac5a24bf7263fade6f80cffe811f49432955e146da3" exitCode=0
Oct 31 13:36:20.989190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:20.989036    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt" event={"ID":"a64a3776-6565-48d0-8c36-5c4d979eeb02","Type":"ContainerDied","Data":"20d158b99f4b4906ebae7ac5a24bf7263fade6f80cffe811f49432955e146da3"}
Oct 31 13:36:21.008629 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.008547    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4071546e-73df-4d16-a000-36fc2372f54b-istio-token" pod "4071546e-73df-4d16-a000-36fc2372f54b" (UID: "4071546e-73df-4d16-a000-36fc2372f54b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.013762 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.013715    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-istio-envoy" pod "4071546e-73df-4d16-a000-36fc2372f54b" (UID: "4071546e-73df-4d16-a000-36fc2372f54b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.014088 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.014058    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4071546e-73df-4d16-a000-36fc2372f54b-kube-api-access-bgclw" pod "4071546e-73df-4d16-a000-36fc2372f54b" (UID: "4071546e-73df-4d16-a000-36fc2372f54b"). InnerVolumeSpecName "kube-api-access-bgclw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.014280 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.014100    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4071546e-73df-4d16-a000-36fc2372f54b-istio-podinfo" pod "4071546e-73df-4d16-a000-36fc2372f54b" (UID: "4071546e-73df-4d16-a000-36fc2372f54b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:36:21.069487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.069450    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4071546e-73df-4d16-a000-36fc2372f54b-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.069487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.069489    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4071546e-73df-4d16-a000-36fc2372f54b-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.069671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.069506    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.069671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.069523    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.069671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.069538    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.069671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.069551    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.069671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.069565    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4071546e-73df-4d16-a000-36fc2372f54b-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.069671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.069579    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgclw\" (UniqueName: \"kubernetes.io/projected/4071546e-73df-4d16-a000-36fc2372f54b-kube-api-access-bgclw\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.069671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.069592    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071546e-73df-4d16-a000-36fc2372f54b-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.214658 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.169687    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:21.265470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.265114    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:36:21.338907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.333934    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:36:21.359295 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.354974    4428 scope.go:122] "RemoveContainer" containerID="f42e6e218cf025c36b4d6b0ebf897e58ccfe565b42aef9fe335c7cbb8fd7d71a"
Oct 31 13:36:21.382231 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:21.368192    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f42e6e218cf025c36b4d6b0ebf897e58ccfe565b42aef9fe335c7cbb8fd7d71a\": container with ID starting with f42e6e218cf025c36b4d6b0ebf897e58ccfe565b42aef9fe335c7cbb8fd7d71a not found: ID does not exist" containerID="f42e6e218cf025c36b4d6b0ebf897e58ccfe565b42aef9fe335c7cbb8fd7d71a"
Oct 31 13:36:21.382231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.368241    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f42e6e218cf025c36b4d6b0ebf897e58ccfe565b42aef9fe335c7cbb8fd7d71a"} err="failed to get container status \"f42e6e218cf025c36b4d6b0ebf897e58ccfe565b42aef9fe335c7cbb8fd7d71a\": rpc error: code = NotFound desc = could not find container \"f42e6e218cf025c36b4d6b0ebf897e58ccfe565b42aef9fe335c7cbb8fd7d71a\": container with ID starting with f42e6e218cf025c36b4d6b0ebf897e58ccfe565b42aef9fe335c7cbb8fd7d71a not found: ID does not exist"
Oct 31 13:36:21.390893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.387096    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:36:21.391296 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.391260    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:21.414110 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.413763    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-podinfo\") pod \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") "
Oct 31 13:36:21.414110 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.413819    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-credential-socket\") pod \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") "
Oct 31 13:36:21.435947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.413940    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-podinfo\") pod \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") "
Oct 31 13:36:21.435947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.414498    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-workload-socket\") pod \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") "
Oct 31 13:36:21.435947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.414622    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-workload-certs\") pod \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") "
Oct 31 13:36:21.435947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.414670    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-token\" (UniqueName: \"kubernetes.io/projected/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-token\") pod \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") "
Oct 31 13:36:21.435947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.414713    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-envoy\") pod \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") "
Oct 31 13:36:21.435947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.414752    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-envoy\") pod \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") "
Oct 31 13:36:21.435947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.414802    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-data\") pod \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") "
Oct 31 13:36:21.435947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.416351    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-credential-socket" pod "9796198f-74dc-491b-b5fe-7ee7bb9ae809" (UID: "9796198f-74dc-491b-b5fe-7ee7bb9ae809"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.435947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.417093    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-workload-socket" pod "af5e815e-eb38-40c4-9651-e4dbbb92dc36" (UID: "af5e815e-eb38-40c4-9651-e4dbbb92dc36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.435947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.417121    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:36:21.435947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.414339    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:36:21.435947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.425888    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-workload-certs" pod "9796198f-74dc-491b-b5fe-7ee7bb9ae809" (UID: "9796198f-74dc-491b-b5fe-7ee7bb9ae809"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.435947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.427242    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-podinfo" pod "af5e815e-eb38-40c4-9651-e4dbbb92dc36" (UID: "af5e815e-eb38-40c4-9651-e4dbbb92dc36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:36:21.435947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.429044    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-token" pod "af5e815e-eb38-40c4-9651-e4dbbb92dc36" (UID: "af5e815e-eb38-40c4-9651-e4dbbb92dc36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.435947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.434184    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-envoy" pod "af5e815e-eb38-40c4-9651-e4dbbb92dc36" (UID: "af5e815e-eb38-40c4-9651-e4dbbb92dc36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.446615 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.437341    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-envoy" pod "9796198f-74dc-491b-b5fe-7ee7bb9ae809" (UID: "9796198f-74dc-491b-b5fe-7ee7bb9ae809"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.446615 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.437549    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"]
Oct 31 13:36:21.446615 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.439341    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-data" pod "9796198f-74dc-491b-b5fe-7ee7bb9ae809" (UID: "9796198f-74dc-491b-b5fe-7ee7bb9ae809"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.449963    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-token\" (UniqueName: \"kubernetes.io/projected/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-token\") pod \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.450515    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93314798-1f48-408a-b87c-d8f809d2f14b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93314798-1f48-408a-b87c-d8f809d2f14b-istiod-ca-cert\") pod \"93314798-1f48-408a-b87c-d8f809d2f14b\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.450625    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-workload-certs\") pod \"a8836a12-5c77-42e2-bb58-e69be67808db\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.450669    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93314798-1f48-408a-b87c-d8f809d2f14b-kube-api-access-b6qj9\" (UniqueName: \"kubernetes.io/projected/93314798-1f48-408a-b87c-d8f809d2f14b-kube-api-access-b6qj9\") pod \"93314798-1f48-408a-b87c-d8f809d2f14b\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.450717    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istiod-ca-cert\") pod \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.450774    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-data\") pod \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.450819    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-workload-socket\") pod \"a8836a12-5c77-42e2-bb58-e69be67808db\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.450882    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93314798-1f48-408a-b87c-d8f809d2f14b-istiod-ca-cert" pod "93314798-1f48-408a-b87c-d8f809d2f14b" (UID: "93314798-1f48-408a-b87c-d8f809d2f14b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.450891    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93314798-1f48-408a-b87c-d8f809d2f14b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93314798-1f48-408a-b87c-d8f809d2f14b-istio-podinfo\") pod \"93314798-1f48-408a-b87c-d8f809d2f14b\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.450939    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-credential-socket\") pod \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.450996    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-credential-socket\") pod \"a8836a12-5c77-42e2-bb58-e69be67808db\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.451133    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-workload-certs\") pod \"69e04722-598c-4608-9742-440e327e389a\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.451227    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8836a12-5c77-42e2-bb58-e69be67808db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8836a12-5c77-42e2-bb58-e69be67808db-istiod-ca-cert\") pod \"a8836a12-5c77-42e2-bb58-e69be67808db\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.451271    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istiod-ca-cert\") pod \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.451323    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-credential-socket" pod "a8836a12-5c77-42e2-bb58-e69be67808db" (UID: "a8836a12-5c77-42e2-bb58-e69be67808db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.451328    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-istio-data\") pod \"a8836a12-5c77-42e2-bb58-e69be67808db\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.451432    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-workload-certs\") pod \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.451532    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9796198f-74dc-491b-b5fe-7ee7bb9ae809-kube-api-access-rslzr\" (UniqueName: \"kubernetes.io/projected/9796198f-74dc-491b-b5fe-7ee7bb9ae809-kube-api-access-rslzr\") pod \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.451548    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-istio-data" pod "a8836a12-5c77-42e2-bb58-e69be67808db" (UID: "a8836a12-5c77-42e2-bb58-e69be67808db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.451566    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e04722-598c-4608-9742-440e327e389a-istio-token\" (UniqueName: \"kubernetes.io/projected/69e04722-598c-4608-9742-440e327e389a-istio-token\") pod \"69e04722-598c-4608-9742-440e327e389a\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.451658    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8836a12-5c77-42e2-bb58-e69be67808db-kube-api-access-7m2bm\" (UniqueName: \"kubernetes.io/projected/a8836a12-5c77-42e2-bb58-e69be67808db-kube-api-access-7m2bm\") pod \"a8836a12-5c77-42e2-bb58-e69be67808db\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.451706    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af5e815e-eb38-40c4-9651-e4dbbb92dc36-kube-api-access-sst8q\" (UniqueName: \"kubernetes.io/projected/af5e815e-eb38-40c4-9651-e4dbbb92dc36-kube-api-access-sst8q\") pod \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\" (UID: \"af5e815e-eb38-40c4-9651-e4dbbb92dc36\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.451738    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-istio-envoy\") pod \"a8836a12-5c77-42e2-bb58-e69be67808db\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.451737    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-workload-certs" pod "69e04722-598c-4608-9742-440e327e389a" (UID: "69e04722-598c-4608-9742-440e327e389a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.451979    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-credential-socket" pod "af5e815e-eb38-40c4-9651-e4dbbb92dc36" (UID: "af5e815e-eb38-40c4-9651-e4dbbb92dc36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.452097    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8836a12-5c77-42e2-bb58-e69be67808db-istiod-ca-cert" pod "a8836a12-5c77-42e2-bb58-e69be67808db" (UID: "a8836a12-5c77-42e2-bb58-e69be67808db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.452185    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-workload-certs" pod "af5e815e-eb38-40c4-9651-e4dbbb92dc36" (UID: "af5e815e-eb38-40c4-9651-e4dbbb92dc36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.452233    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-workload-socket\") pod \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\" (UID: \"9796198f-74dc-491b-b5fe-7ee7bb9ae809\") "
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.453108    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-token" pod "9796198f-74dc-491b-b5fe-7ee7bb9ae809" (UID: "9796198f-74dc-491b-b5fe-7ee7bb9ae809"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.453438    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istiod-ca-cert" pod "af5e815e-eb38-40c4-9651-e4dbbb92dc36" (UID: "af5e815e-eb38-40c4-9651-e4dbbb92dc36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:21.454063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.453653    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-workload-certs" pod "a8836a12-5c77-42e2-bb58-e69be67808db" (UID: "a8836a12-5c77-42e2-bb58-e69be67808db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.454254    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-data" pod "af5e815e-eb38-40c4-9651-e4dbbb92dc36" (UID: "af5e815e-eb38-40c4-9651-e4dbbb92dc36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.454285    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-workload-socket" pod "9796198f-74dc-491b-b5fe-7ee7bb9ae809" (UID: "9796198f-74dc-491b-b5fe-7ee7bb9ae809"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.454449    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-workload-socket" pod "a8836a12-5c77-42e2-bb58-e69be67808db" (UID: "a8836a12-5c77-42e2-bb58-e69be67808db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455018    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455058    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93314798-1f48-408a-b87c-d8f809d2f14b-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455060    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istiod-ca-cert" pod "9796198f-74dc-491b-b5fe-7ee7bb9ae809" (UID: "9796198f-74dc-491b-b5fe-7ee7bb9ae809"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455087    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455110    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455125    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455140    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455158    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455177    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455195    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455210    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8836a12-5c77-42e2-bb58-e69be67808db-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455228    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455248    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455263    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455278    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455298    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455314    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455330    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455345    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455365    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af5e815e-eb38-40c4-9651-e4dbbb92dc36-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455381    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.456499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.455396    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.458338 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.458216    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93314798-1f48-408a-b87c-d8f809d2f14b-istio-podinfo" pod "93314798-1f48-408a-b87c-d8f809d2f14b" (UID: "93314798-1f48-408a-b87c-d8f809d2f14b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:36:21.459135 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.458433    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93314798-1f48-408a-b87c-d8f809d2f14b-kube-api-access-b6qj9" pod "93314798-1f48-408a-b87c-d8f809d2f14b" (UID: "93314798-1f48-408a-b87c-d8f809d2f14b"). InnerVolumeSpecName "kube-api-access-b6qj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.460242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.460156    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9796198f-74dc-491b-b5fe-7ee7bb9ae809-kube-api-access-rslzr" pod "9796198f-74dc-491b-b5fe-7ee7bb9ae809" (UID: "9796198f-74dc-491b-b5fe-7ee7bb9ae809"). InnerVolumeSpecName "kube-api-access-rslzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.464537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.461393    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5e815e-eb38-40c4-9651-e4dbbb92dc36-kube-api-access-sst8q" pod "af5e815e-eb38-40c4-9651-e4dbbb92dc36" (UID: "af5e815e-eb38-40c4-9651-e4dbbb92dc36"). InnerVolumeSpecName "kube-api-access-sst8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.464537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.463748    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-istio-envoy" pod "a8836a12-5c77-42e2-bb58-e69be67808db" (UID: "a8836a12-5c77-42e2-bb58-e69be67808db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.464816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.464706    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-podinfo" pod "9796198f-74dc-491b-b5fe-7ee7bb9ae809" (UID: "9796198f-74dc-491b-b5fe-7ee7bb9ae809"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:36:21.468739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.465595    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e04722-598c-4608-9742-440e327e389a-istio-token" pod "69e04722-598c-4608-9742-440e327e389a" (UID: "69e04722-598c-4608-9742-440e327e389a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.470819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.470770    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"]
Oct 31 13:36:21.487113 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.487085    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8836a12-5c77-42e2-bb58-e69be67808db-kube-api-access-7m2bm" pod "a8836a12-5c77-42e2-bb58-e69be67808db" (UID: "a8836a12-5c77-42e2-bb58-e69be67808db"). InnerVolumeSpecName "kube-api-access-7m2bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.490491 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.487893    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:36:21.496907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.496730    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:36:21.497286 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.497230    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-h6l58"]
Oct 31 13:36:21.499301 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.499242    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.556435    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-workload-certs\") pod \"a64a3776-6565-48d0-8c36-5c4d979eeb02\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.556489    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e04722-598c-4608-9742-440e327e389a-kube-api-access-q6bn6\" (UniqueName: \"kubernetes.io/projected/69e04722-598c-4608-9742-440e327e389a-kube-api-access-q6bn6\") pod \"69e04722-598c-4608-9742-440e327e389a\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.556520    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8571392-d978-4512-939b-69b9fadb6734-kube-api-access-bktkl\" (UniqueName: \"kubernetes.io/projected/b8571392-d978-4512-939b-69b9fadb6734-kube-api-access-bktkl\") pod \"b8571392-d978-4512-939b-69b9fadb6734\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.556552    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8836a12-5c77-42e2-bb58-e69be67808db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8836a12-5c77-42e2-bb58-e69be67808db-istio-podinfo\") pod \"a8836a12-5c77-42e2-bb58-e69be67808db\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.556584    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-istio-envoy\") pod \"69e04722-598c-4608-9742-440e327e389a\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.556614    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-credential-socket\") pod \"a64a3776-6565-48d0-8c36-5c4d979eeb02\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.556645    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-workload-socket\") pod \"69e04722-598c-4608-9742-440e327e389a\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.556673    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-credential-socket\") pod \"b8571392-d978-4512-939b-69b9fadb6734\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.556707    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-credential-socket\") pod \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.556736    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8836a12-5c77-42e2-bb58-e69be67808db-istio-token\" (UniqueName: \"kubernetes.io/projected/a8836a12-5c77-42e2-bb58-e69be67808db-istio-token\") pod \"a8836a12-5c77-42e2-bb58-e69be67808db\" (UID: \"a8836a12-5c77-42e2-bb58-e69be67808db\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.556772    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8571392-d978-4512-939b-69b9fadb6734-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8571392-d978-4512-939b-69b9fadb6734-istiod-ca-cert\") pod \"b8571392-d978-4512-939b-69b9fadb6734\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.556835    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-data\") pod \"a64a3776-6565-48d0-8c36-5c4d979eeb02\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.556922    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-workload-socket\") pod \"b8571392-d978-4512-939b-69b9fadb6734\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.556973    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69e04722-598c-4608-9742-440e327e389a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69e04722-598c-4608-9742-440e327e389a-istio-podinfo\") pod \"69e04722-598c-4608-9742-440e327e389a\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557009    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-envoy\") pod \"a64a3776-6565-48d0-8c36-5c4d979eeb02\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557066    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93314798-1f48-408a-b87c-d8f809d2f14b-istio-token\" (UniqueName: \"kubernetes.io/projected/93314798-1f48-408a-b87c-d8f809d2f14b-istio-token\") pod \"93314798-1f48-408a-b87c-d8f809d2f14b\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557096    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-istio-data\") pod \"69e04722-598c-4608-9742-440e327e389a\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557129    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-credential-socket\") pod \"93314798-1f48-408a-b87c-d8f809d2f14b\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557174    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-data\") pod \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557195    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-workload-certs\") pod \"b8571392-d978-4512-939b-69b9fadb6734\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557211    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-istio-envoy\") pod \"b8571392-d978-4512-939b-69b9fadb6734\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557231    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a64a3776-6565-48d0-8c36-5c4d979eeb02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a64a3776-6565-48d0-8c36-5c4d979eeb02-istiod-ca-cert\") pod \"a64a3776-6565-48d0-8c36-5c4d979eeb02\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557274    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a64a3776-6565-48d0-8c36-5c4d979eeb02-kube-api-access-l67qg\" (UniqueName: \"kubernetes.io/projected/a64a3776-6565-48d0-8c36-5c4d979eeb02-kube-api-access-l67qg\") pod \"a64a3776-6565-48d0-8c36-5c4d979eeb02\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557322    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-workload-socket\") pod \"a64a3776-6565-48d0-8c36-5c4d979eeb02\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557337    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8571392-d978-4512-939b-69b9fadb6734-istio-token\" (UniqueName: \"kubernetes.io/projected/b8571392-d978-4512-939b-69b9fadb6734-istio-token\") pod \"b8571392-d978-4512-939b-69b9fadb6734\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557415    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-envoy\") pod \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557449    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-workload-socket\") pod \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557566    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaab76aa-bfe5-4413-83a8-bc1037c2b525-kube-api-access-lg9jt\" (UniqueName: \"kubernetes.io/projected/aaab76aa-bfe5-4413-83a8-bc1037c2b525-kube-api-access-lg9jt\") pod \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557603    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-podinfo\") pod \"a64a3776-6565-48d0-8c36-5c4d979eeb02\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557634    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-token\" (UniqueName: \"kubernetes.io/projected/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-token\") pod \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557751    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-istio-data\") pod \"93314798-1f48-408a-b87c-d8f809d2f14b\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557782    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-credential-socket\") pod \"69e04722-598c-4608-9742-440e327e389a\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557826    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-workload-certs\") pod \"93314798-1f48-408a-b87c-d8f809d2f14b\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557855    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-podinfo\") pod \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.557984    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-workload-socket\") pod \"93314798-1f48-408a-b87c-d8f809d2f14b\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.558014    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e04722-598c-4608-9742-440e327e389a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69e04722-598c-4608-9742-440e327e389a-istiod-ca-cert\") pod \"69e04722-598c-4608-9742-440e327e389a\" (UID: \"69e04722-598c-4608-9742-440e327e389a\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.558054    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-token\" (UniqueName: \"kubernetes.io/projected/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-token\") pod \"a64a3776-6565-48d0-8c36-5c4d979eeb02\" (UID: \"a64a3776-6565-48d0-8c36-5c4d979eeb02\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.558091    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istiod-ca-cert\") pod \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.558125    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8571392-d978-4512-939b-69b9fadb6734-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8571392-d978-4512-939b-69b9fadb6734-istio-podinfo\") pod \"b8571392-d978-4512-939b-69b9fadb6734\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.558159    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-workload-certs\") pod \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\" (UID: \"aaab76aa-bfe5-4413-83a8-bc1037c2b525\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.558199    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-istio-envoy\") pod \"93314798-1f48-408a-b87c-d8f809d2f14b\" (UID: \"93314798-1f48-408a-b87c-d8f809d2f14b\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.558240    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-istio-data\") pod \"b8571392-d978-4512-939b-69b9fadb6734\" (UID: \"b8571392-d978-4512-939b-69b9fadb6734\") "
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.559188    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6qj9\" (UniqueName: \"kubernetes.io/projected/93314798-1f48-408a-b87c-d8f809d2f14b-kube-api-access-b6qj9\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.559213    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93314798-1f48-408a-b87c-d8f809d2f14b-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.559227    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.559240    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rslzr\" (UniqueName: \"kubernetes.io/projected/9796198f-74dc-491b-b5fe-7ee7bb9ae809-kube-api-access-rslzr\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.559255    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69e04722-598c-4608-9742-440e327e389a-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.559268    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m2bm\" (UniqueName: \"kubernetes.io/projected/a8836a12-5c77-42e2-bb58-e69be67808db-kube-api-access-7m2bm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.559281    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sst8q\" (UniqueName: \"kubernetes.io/projected/af5e815e-eb38-40c4-9651-e4dbbb92dc36-kube-api-access-sst8q\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.559295    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8836a12-5c77-42e2-bb58-e69be67808db-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.559528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.559310    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9796198f-74dc-491b-b5fe-7ee7bb9ae809-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.562449 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.562250    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-istio-data" pod "b8571392-d978-4512-939b-69b9fadb6734" (UID: "b8571392-d978-4512-939b-69b9fadb6734"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.562915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.562597    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-workload-certs" pod "a64a3776-6565-48d0-8c36-5c4d979eeb02" (UID: "a64a3776-6565-48d0-8c36-5c4d979eeb02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.563563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.563542    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-data" pod "a64a3776-6565-48d0-8c36-5c4d979eeb02" (UID: "a64a3776-6565-48d0-8c36-5c4d979eeb02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.563719 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.563659    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-workload-socket" pod "aaab76aa-bfe5-4413-83a8-bc1037c2b525" (UID: "aaab76aa-bfe5-4413-83a8-bc1037c2b525"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.564184 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.564163    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-credential-socket" pod "69e04722-598c-4608-9742-440e327e389a" (UID: "69e04722-598c-4608-9742-440e327e389a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.564672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.564644    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-workload-socket" pod "69e04722-598c-4608-9742-440e327e389a" (UID: "69e04722-598c-4608-9742-440e327e389a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.564818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.564788    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a64a3776-6565-48d0-8c36-5c4d979eeb02-istiod-ca-cert" pod "a64a3776-6565-48d0-8c36-5c4d979eeb02" (UID: "a64a3776-6565-48d0-8c36-5c4d979eeb02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:21.565097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.565073    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-workload-socket" pod "b8571392-d978-4512-939b-69b9fadb6734" (UID: "b8571392-d978-4512-939b-69b9fadb6734"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.569781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.569759    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-workload-socket" pod "a64a3776-6565-48d0-8c36-5c4d979eeb02" (UID: "a64a3776-6565-48d0-8c36-5c4d979eeb02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.569954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.569934    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-workload-certs" pod "93314798-1f48-408a-b87c-d8f809d2f14b" (UID: "93314798-1f48-408a-b87c-d8f809d2f14b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.570197 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.570176    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaab76aa-bfe5-4413-83a8-bc1037c2b525-kube-api-access-lg9jt" pod "aaab76aa-bfe5-4413-83a8-bc1037c2b525" (UID: "aaab76aa-bfe5-4413-83a8-bc1037c2b525"). InnerVolumeSpecName "kube-api-access-lg9jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.570338 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.570321    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69e04722-598c-4608-9742-440e327e389a-istio-podinfo" pod "69e04722-598c-4608-9742-440e327e389a" (UID: "69e04722-598c-4608-9742-440e327e389a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:36:21.570675 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.570640    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8836a12-5c77-42e2-bb58-e69be67808db-istio-podinfo" pod "a8836a12-5c77-42e2-bb58-e69be67808db" (UID: "a8836a12-5c77-42e2-bb58-e69be67808db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:36:21.570899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.570753    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-workload-socket" pod "93314798-1f48-408a-b87c-d8f809d2f14b" (UID: "93314798-1f48-408a-b87c-d8f809d2f14b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.571127 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.571079    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e04722-598c-4608-9742-440e327e389a-istiod-ca-cert" pod "69e04722-598c-4608-9742-440e327e389a" (UID: "69e04722-598c-4608-9742-440e327e389a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:21.572485 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.571255    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-credential-socket" pod "a64a3776-6565-48d0-8c36-5c4d979eeb02" (UID: "a64a3776-6565-48d0-8c36-5c4d979eeb02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.572485 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.571468    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-data" pod "aaab76aa-bfe5-4413-83a8-bc1037c2b525" (UID: "aaab76aa-bfe5-4413-83a8-bc1037c2b525"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.572485 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.571748    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-istio-data" pod "69e04722-598c-4608-9742-440e327e389a" (UID: "69e04722-598c-4608-9742-440e327e389a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.572485 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.571952    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-credential-socket" pod "93314798-1f48-408a-b87c-d8f809d2f14b" (UID: "93314798-1f48-408a-b87c-d8f809d2f14b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.573278 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.573027    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-istio-data" pod "93314798-1f48-408a-b87c-d8f809d2f14b" (UID: "93314798-1f48-408a-b87c-d8f809d2f14b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.575116 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.575083    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-workload-certs" pod "b8571392-d978-4512-939b-69b9fadb6734" (UID: "b8571392-d978-4512-939b-69b9fadb6734"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.576339 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.576145    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-istio-envoy" pod "b8571392-d978-4512-939b-69b9fadb6734" (UID: "b8571392-d978-4512-939b-69b9fadb6734"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.576663 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.576548    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-workload-certs" pod "aaab76aa-bfe5-4413-83a8-bc1037c2b525" (UID: "aaab76aa-bfe5-4413-83a8-bc1037c2b525"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.577184 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.576999    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-credential-socket" pod "aaab76aa-bfe5-4413-83a8-bc1037c2b525" (UID: "aaab76aa-bfe5-4413-83a8-bc1037c2b525"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.577630 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.577344    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-credential-socket" pod "b8571392-d978-4512-939b-69b9fadb6734" (UID: "b8571392-d978-4512-939b-69b9fadb6734"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.578369 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.577834    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8571392-d978-4512-939b-69b9fadb6734-istio-podinfo" pod "b8571392-d978-4512-939b-69b9fadb6734" (UID: "b8571392-d978-4512-939b-69b9fadb6734"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:36:21.578369 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.577927    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istiod-ca-cert" pod "aaab76aa-bfe5-4413-83a8-bc1037c2b525" (UID: "aaab76aa-bfe5-4413-83a8-bc1037c2b525"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:21.580932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.580831    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-istio-envoy" pod "93314798-1f48-408a-b87c-d8f809d2f14b" (UID: "93314798-1f48-408a-b87c-d8f809d2f14b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.581356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.581293    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8571392-d978-4512-939b-69b9fadb6734-istiod-ca-cert" pod "b8571392-d978-4512-939b-69b9fadb6734" (UID: "b8571392-d978-4512-939b-69b9fadb6734"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:21.582999 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.582720    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8836a12-5c77-42e2-bb58-e69be67808db-istio-token" pod "a8836a12-5c77-42e2-bb58-e69be67808db" (UID: "a8836a12-5c77-42e2-bb58-e69be67808db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.584912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.583128    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:21.586254 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.586208    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-podinfo" pod "a64a3776-6565-48d0-8c36-5c4d979eeb02" (UID: "a64a3776-6565-48d0-8c36-5c4d979eeb02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:36:21.593226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.593188    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-envoy" pod "a64a3776-6565-48d0-8c36-5c4d979eeb02" (UID: "a64a3776-6565-48d0-8c36-5c4d979eeb02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.593321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.593225    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-envoy" pod "aaab76aa-bfe5-4413-83a8-bc1037c2b525" (UID: "aaab76aa-bfe5-4413-83a8-bc1037c2b525"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.593321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.593249    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-token" pod "aaab76aa-bfe5-4413-83a8-bc1037c2b525" (UID: "aaab76aa-bfe5-4413-83a8-bc1037c2b525"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.593451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.593329    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93314798-1f48-408a-b87c-d8f809d2f14b-istio-token" pod "93314798-1f48-408a-b87c-d8f809d2f14b" (UID: "93314798-1f48-408a-b87c-d8f809d2f14b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.593613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.593588    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-podinfo" pod "aaab76aa-bfe5-4413-83a8-bc1037c2b525" (UID: "aaab76aa-bfe5-4413-83a8-bc1037c2b525"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:36:21.593730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.593613    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-istio-envoy" pod "69e04722-598c-4608-9742-440e327e389a" (UID: "69e04722-598c-4608-9742-440e327e389a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:21.593730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.593696    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e04722-598c-4608-9742-440e327e389a-kube-api-access-q6bn6" pod "69e04722-598c-4608-9742-440e327e389a" (UID: "69e04722-598c-4608-9742-440e327e389a"). InnerVolumeSpecName "kube-api-access-q6bn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.593996 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.593744    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64a3776-6565-48d0-8c36-5c4d979eeb02-kube-api-access-l67qg" pod "a64a3776-6565-48d0-8c36-5c4d979eeb02" (UID: "a64a3776-6565-48d0-8c36-5c4d979eeb02"). InnerVolumeSpecName "kube-api-access-l67qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.593996 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.593819    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8571392-d978-4512-939b-69b9fadb6734-istio-token" pod "b8571392-d978-4512-939b-69b9fadb6734" (UID: "b8571392-d978-4512-939b-69b9fadb6734"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.596901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.596862    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-token" pod "a64a3776-6565-48d0-8c36-5c4d979eeb02" (UID: "a64a3776-6565-48d0-8c36-5c4d979eeb02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.599089 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.599064    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8571392-d978-4512-939b-69b9fadb6734-kube-api-access-bktkl" pod "b8571392-d978-4512-939b-69b9fadb6734" (UID: "b8571392-d978-4512-939b-69b9fadb6734"). InnerVolumeSpecName "kube-api-access-bktkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:21.666050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666018    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666049    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69e04722-598c-4608-9742-440e327e389a-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666063    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666078    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666090    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8571392-d978-4512-939b-69b9fadb6734-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666104    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666116    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666126    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666138    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666155    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6bn6\" (UniqueName: \"kubernetes.io/projected/69e04722-598c-4608-9742-440e327e389a-kube-api-access-q6bn6\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666168    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bktkl\" (UniqueName: \"kubernetes.io/projected/b8571392-d978-4512-939b-69b9fadb6734-kube-api-access-bktkl\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666179    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8836a12-5c77-42e2-bb58-e69be67808db-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666193    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666206    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666218    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666233    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666247    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666258    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8836a12-5c77-42e2-bb58-e69be67808db-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666271    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8571392-d978-4512-939b-69b9fadb6734-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666282    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666295    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666310    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69e04722-598c-4608-9742-440e327e389a-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666323    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666337    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93314798-1f48-408a-b87c-d8f809d2f14b-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666350    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666363    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666376    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666390    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666405    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8571392-d978-4512-939b-69b9fadb6734-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666419    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a64a3776-6565-48d0-8c36-5c4d979eeb02-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666434    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l67qg\" (UniqueName: \"kubernetes.io/projected/a64a3776-6565-48d0-8c36-5c4d979eeb02-kube-api-access-l67qg\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666447    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64a3776-6565-48d0-8c36-5c4d979eeb02-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666461    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8571392-d978-4512-939b-69b9fadb6734-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666474    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666485    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaab76aa-bfe5-4413-83a8-bc1037c2b525-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666499    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg9jt\" (UniqueName: \"kubernetes.io/projected/aaab76aa-bfe5-4413-83a8-bc1037c2b525-kube-api-access-lg9jt\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666513    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a64a3776-6565-48d0-8c36-5c4d979eeb02-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666525    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666538    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666550    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e04722-598c-4608-9742-440e327e389a-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.666539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666563    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93314798-1f48-408a-b87c-d8f809d2f14b-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.669334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.666577    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaab76aa-bfe5-4413-83a8-bc1037c2b525-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:21.755035 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.754860    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:21.929893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:21.929463    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:22.065959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.053337    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns" event={"ID":"93314798-1f48-408a-b87c-d8f809d2f14b","Type":"ContainerDied","Data":"6af5a27412fa8cd8a089c7a6144d6659f134566beb1f392c1b0d9162a29db245"}
Oct 31 13:36:22.065959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.053406    4428 scope.go:122] "RemoveContainer" containerID="efa22e7ed2f6b7bf67c47aba76e075d62775b017722de4af5944aa9e1545e58f"
Oct 31 13:36:22.065959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.053531    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"
Oct 31 13:36:22.110042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.106421    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"
Oct 31 13:36:22.116604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.116555    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn" event={"ID":"69e04722-598c-4608-9742-440e327e389a","Type":"ContainerDied","Data":"7b340b823bad6d5d700a1b1e5038c1d2ccaa8770424113065ee01990f70bb27c"}
Oct 31 13:36:22.144919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.141446    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx" event={"ID":"aaab76aa-bfe5-4413-83a8-bc1037c2b525","Type":"ContainerDied","Data":"a50c69ec23b1b82efd159356d6c135121f270c5623b213fed235276ebb1f0f84"}
Oct 31 13:36:22.144919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.141587    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"
Oct 31 13:36:22.148661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.148485    4428 scope.go:122] "RemoveContainer" containerID="3a3470b0c35f73c4ef5c10cbe6cadf696674bebc1230b9fe1d147f3791082c1f"
Oct 31 13:36:22.151759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.151700    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd" event={"ID":"a8836a12-5c77-42e2-bb58-e69be67808db","Type":"ContainerDied","Data":"492cd06048c11ab6dc3a8d2e68baa6c7314578c7ea711ff24714f56cb09ad609"}
Oct 31 13:36:22.151896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.151817    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"
Oct 31 13:36:22.160592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.158117    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt" event={"ID":"a64a3776-6565-48d0-8c36-5c4d979eeb02","Type":"ContainerDied","Data":"e434e5e14b6269a2c9ff8247e3fc1371599e919ca5d0c18d3509423839056594"}
Oct 31 13:36:22.160592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.158226    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"
Oct 31 13:36:22.161956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.160845    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj" event={"ID":"af5e815e-eb38-40c4-9651-e4dbbb92dc36","Type":"ContainerDied","Data":"7f72ce4e9b19f8f78a9ce5b76c45303e94b99074cb94288e0b0ff6e1e12d535c"}
Oct 31 13:36:22.161956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.160978    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"
Oct 31 13:36:22.162907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.162856    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2" event={"ID":"9796198f-74dc-491b-b5fe-7ee7bb9ae809","Type":"ContainerDied","Data":"b4dcdf86f96907844d8e5aaa3ff08034c839ff3c9ff9bce019a813222a4c1cdc"}
Oct 31 13:36:22.163029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.162989    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"
Oct 31 13:36:22.188789 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.188737    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"]
Oct 31 13:36:22.196350 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.196319    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-cbknn"]
Oct 31 13:36:22.198773 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.198741    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s" event={"ID":"b8571392-d978-4512-939b-69b9fadb6734","Type":"ContainerDied","Data":"bdcb897decb9197fd920fd74b796639502605b2ede2de4c60907d2373ca587bb"}
Oct 31 13:36:22.199273 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.198819    4428 scope.go:122] "RemoveContainer" containerID="bdee988fff66234b73085f7d21cb3c50213f9a09b6e07328bb1720a57d4f3fab"
Oct 31 13:36:22.199273 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.198980    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"
Oct 31 13:36:22.244119 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.244080    4428 scope.go:122] "RemoveContainer" containerID="8f11954f00bcb576c130524e86f1593ea416ce35774aff28da5f3613f54b13f7"
Oct 31 13:36:22.257831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.257791    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"]
Oct 31 13:36:22.324968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.320558    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-l52ns"]
Oct 31 13:36:22.472048 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.456400    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:22.539165 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.475787    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"]
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539500    4428 cpu_manager.go:414] "containerMap: removing container" podUID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539530    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539587    4428 cpu_manager.go:414] "containerMap: removing container" podUID="93314798-1f48-408a-b87c-d8f809d2f14b" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539597    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93314798-1f48-408a-b87c-d8f809d2f14b" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539609    4428 cpu_manager.go:414] "containerMap: removing container" podUID="c934f1a7-3712-4955-ad79-5505b3760938" containerName="revoke-keys"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539617    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c934f1a7-3712-4955-ad79-5505b3760938" containerName="revoke-keys"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539695    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9796198f-74dc-491b-b5fe-7ee7bb9ae809" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539705    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9796198f-74dc-491b-b5fe-7ee7bb9ae809" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539713    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539721    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539746    4428 cpu_manager.go:414] "containerMap: removing container" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539754    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539787    4428 cpu_manager.go:414] "containerMap: removing container" podUID="69e04722-598c-4608-9742-440e327e389a" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539794    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e04722-598c-4608-9742-440e327e389a" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539804    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539811    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539827    4428 cpu_manager.go:414] "containerMap: removing container" podUID="a64a3776-6565-48d0-8c36-5c4d979eeb02" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539836    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a64a3776-6565-48d0-8c36-5c4d979eeb02" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539853    4428 cpu_manager.go:414] "containerMap: removing container" podUID="aaab76aa-bfe5-4413-83a8-bc1037c2b525" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.539887    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaab76aa-bfe5-4413-83a8-bc1037c2b525" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.540168    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="93314798-1f48-408a-b87c-d8f809d2f14b" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.540189    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c934f1a7-3712-4955-ad79-5505b3760938" containerName="revoke-keys"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.540240    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9796198f-74dc-491b-b5fe-7ee7bb9ae809" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.540250    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4071546e-73df-4d16-a000-36fc2372f54b" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.540263    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.540276    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e04722-598c-4608-9742-440e327e389a" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.540285    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8571392-d978-4512-939b-69b9fadb6734" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.540296    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a64a3776-6565-48d0-8c36-5c4d979eeb02" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.540305    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaab76aa-bfe5-4413-83a8-bc1037c2b525" containerName="istio-proxy"
Oct 31 13:36:22.613972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.540314    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" containerName="istio-proxy"
Oct 31 13:36:22.646075 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.645996    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"]
Oct 31 13:36:22.646371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.646324    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"]
Oct 31 13:36:22.646605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.646578    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.656034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.655992    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-76c4s"]
Oct 31 13:36:22.687196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.687165    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"]
Oct 31 13:36:22.698430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.696245    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-5e8f29-openshift-default-dockercfg-m5586\""
Oct 31 13:36:22.756630 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.712014    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-v8bk2"]
Oct 31 13:36:22.756630 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.723460    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"]
Oct 31 13:36:22.757063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.757033    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-828jt"]
Oct 31 13:36:22.773931 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.764169    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:22.824861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.818625    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b290805f-2d02-4d45-abbd-d0f0f8afe535-istiod-ca-cert\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.824861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.818696    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf2ld\" (UniqueName: \"kubernetes.io/projected/b290805f-2d02-4d45-abbd-d0f0f8afe535-kube-api-access-wf2ld\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.824861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.818735    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-workload-certs\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.824861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.818801    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-credential-socket\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.824861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.818840    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-workload-socket\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.834192 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.825456    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-data\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.834192 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.825525    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-envoy\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.834192 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.825566    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-token\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.834192 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.825715    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-podinfo\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.899460 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.899426    4428 scope.go:122] "RemoveContainer" containerID="20d158b99f4b4906ebae7ac5a24bf7263fade6f80cffe811f49432955e146da3"
Oct 31 13:36:22.931265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.931233    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b290805f-2d02-4d45-abbd-d0f0f8afe535-istiod-ca-cert\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.938125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.938092    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf2ld\" (UniqueName: \"kubernetes.io/projected/b290805f-2d02-4d45-abbd-d0f0f8afe535-kube-api-access-wf2ld\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.938376 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.938358    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-workload-certs\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.938568 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.938551    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-credential-socket\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.940984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.940959    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-workload-socket\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.941206 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.941188    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-data\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.941344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.941330    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-envoy\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.941471 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.941457    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-token\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.941712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.941692    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-podinfo\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.947998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.947970    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-credential-socket\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.948692 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.948663    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b290805f-2d02-4d45-abbd-d0f0f8afe535-istiod-ca-cert\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.949384 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.949353    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-workload-certs\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.949674 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.949647    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-data\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.949957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.949928    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-workload-socket\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.953159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.953118    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-envoy\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.967022 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.966997    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4071546e-73df-4d16-a000-36fc2372f54b" path="/var/lib/kubelet/pods/4071546e-73df-4d16-a000-36fc2372f54b/volumes"
Oct 31 13:36:22.968089 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.968070    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69e04722-598c-4608-9742-440e327e389a" path="/var/lib/kubelet/pods/69e04722-598c-4608-9742-440e327e389a/volumes"
Oct 31 13:36:22.968733 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.968703    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-token\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.969085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.969059    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf2ld\" (UniqueName: \"kubernetes.io/projected/b290805f-2d02-4d45-abbd-d0f0f8afe535-kube-api-access-wf2ld\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.969231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.969150    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93314798-1f48-408a-b87c-d8f809d2f14b" path="/var/lib/kubelet/pods/93314798-1f48-408a-b87c-d8f809d2f14b/volumes"
Oct 31 13:36:22.971338 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.971307    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9796198f-74dc-491b-b5fe-7ee7bb9ae809" path="/var/lib/kubelet/pods/9796198f-74dc-491b-b5fe-7ee7bb9ae809/volumes"
Oct 31 13:36:22.971433 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.971369    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-podinfo\") pod \"e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") " pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:22.971817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.971797    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a64a3776-6565-48d0-8c36-5c4d979eeb02" path="/var/lib/kubelet/pods/a64a3776-6565-48d0-8c36-5c4d979eeb02/volumes"
Oct 31 13:36:22.972338 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.972320    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8571392-d978-4512-939b-69b9fadb6734" path="/var/lib/kubelet/pods/b8571392-d978-4512-939b-69b9fadb6734/volumes"
Oct 31 13:36:22.974278 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.974253    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"]
Oct 31 13:36:22.974367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.974346    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-bz4kj"]
Oct 31 13:36:22.974424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.974370    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"]
Oct 31 13:36:22.974424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.974386    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-zc7pd"]
Oct 31 13:36:22.974424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.974401    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"]
Oct 31 13:36:22.974424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.974419    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-j5hjx"]
Oct 31 13:36:22.995796 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:22.995742    4428 scope.go:122] "RemoveContainer" containerID="3c0f0842e7377dd501d6d955f7aab95cbba47bf1f6a3ea579bad79d33b4b773a"
Oct 31 13:36:23.041096 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:23.034309    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:23.088046 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:23.087984    4428 scope.go:122] "RemoveContainer" containerID="d171285953525cf3a208e44c8fba973804fa5563b3413e5785d39cb6f371afa7"
Oct 31 13:36:23.091448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:23.091426    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:23.115268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:23.115243    4428 scope.go:122] "RemoveContainer" containerID="663b7fae31d3997e9a3aa2e3842bac417d1b7aa1223a719ba0603aa3645d1645"
Oct 31 13:36:23.399752 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:36:23.399686    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb290805f_2d02_4d45_abbd_d0f0f8afe535.slice/crio-80e8ec983302e5855efe30f113ef19f25b71840e916ec16da77908b62a2ad4cc WatchSource:0}: Error finding container 80e8ec983302e5855efe30f113ef19f25b71840e916ec16da77908b62a2ad4cc: Status 404 returned error can't find the container with id 80e8ec983302e5855efe30f113ef19f25b71840e916ec16da77908b62a2ad4cc
Oct 31 13:36:23.401594 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:23.401568    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"]
Oct 31 13:36:23.403627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:23.403586    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:23.403717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:23.403670    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:23.403785 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:23.403714    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:23.888343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:23.888311    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:24.065539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:24.065503    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:24.232076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:24.232039    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb" event={"ID":"b290805f-2d02-4d45-abbd-d0f0f8afe535","Type":"ContainerStarted","Data":"2e621b67784052dd20cecfbcbb2092d3c74843eae22e2a62a4f98e6b7f109989"}
Oct 31 13:36:24.232247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:24.232096    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb" event={"ID":"b290805f-2d02-4d45-abbd-d0f0f8afe535","Type":"ContainerStarted","Data":"80e8ec983302e5855efe30f113ef19f25b71840e916ec16da77908b62a2ad4cc"}
Oct 31 13:36:24.259755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:24.259695    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb" podStartSLOduration=2.259678782 podStartE2EDuration="2.259678782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:24.257205691 +0000 UTC m=+1981.655554317" watchObservedRunningTime="2026-10-31 13:36:24.259678782 +0000 UTC m=+1981.658027406"
Oct 31 13:36:24.262337 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:24.262308    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:24.540100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:24.540015    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"
Oct 31 13:36:24.760674 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:24.760632    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8836a12-5c77-42e2-bb58-e69be67808db" path="/var/lib/kubelet/pods/a8836a12-5c77-42e2-bb58-e69be67808db/volumes"
Oct 31 13:36:24.761490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:24.761469    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaab76aa-bfe5-4413-83a8-bc1037c2b525" path="/var/lib/kubelet/pods/aaab76aa-bfe5-4413-83a8-bc1037c2b525/volumes"
Oct 31 13:36:24.762219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:24.762199    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af5e815e-eb38-40c4-9651-e4dbbb92dc36" path="/var/lib/kubelet/pods/af5e815e-eb38-40c4-9651-e4dbbb92dc36/volumes"
Oct 31 13:36:24.911010 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:24.910140    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:25.092378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:25.092347    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:25.098136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:25.098112    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:25.240016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:25.239984    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:25.241092 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:25.241071    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:36:25.945337 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:25.945301    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:26.167898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.167538    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:26.340949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.339681    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"]
Oct 31 13:36:26.359437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.359405    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.363918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.362405    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"]
Oct 31 13:36:26.438922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.438847    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9" podUID="e34f0c7f-398d-4614-8851-913948829895" containerName="payload-pre-processing" containerID="cri-o://1be9760b4fca4aa57c346eaae358b936f53cc5a718d7e515f41e2688cb861d1f" gracePeriod=25
Oct 31 13:36:26.474788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.474761    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.474912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.474817    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.475040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.475014    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.475104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.475063    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab494491-9445-49af-935d-f33df1c6589b-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.475178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.475110    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab494491-9445-49af-935d-f33df1c6589b-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.475236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.475175    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.475236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.475211    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.475426 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.475410    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab494491-9445-49af-935d-f33df1c6589b-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.475515 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.475463    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw8lk\" (UniqueName: \"kubernetes.io/projected/ab494491-9445-49af-935d-f33df1c6589b-kube-api-access-mw8lk\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.493518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.493497    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"]
Oct 31 13:36:26.538755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.530709    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-752869-xl89g"]
Oct 31 13:36:26.538755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.535593    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:26.597124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.595688    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.597124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.595759    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.597124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.595830    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.597124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.595861    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab494491-9445-49af-935d-f33df1c6589b-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.597124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.595903    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab494491-9445-49af-935d-f33df1c6589b-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.597124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.595951    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.597124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.595988    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.597124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.596031    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab494491-9445-49af-935d-f33df1c6589b-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.597124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.596071    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw8lk\" (UniqueName: \"kubernetes.io/projected/ab494491-9445-49af-935d-f33df1c6589b-kube-api-access-mw8lk\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.598367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.596854    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.598891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.598843    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.599395 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.599348    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.623260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.622469    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab494491-9445-49af-935d-f33df1c6589b-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.623260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.622849    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.630300 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.630262    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"]
Oct 31 13:36:26.641226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.634678    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw8lk\" (UniqueName: \"kubernetes.io/projected/ab494491-9445-49af-935d-f33df1c6589b-kube-api-access-mw8lk\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.655507 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.646801    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab494491-9445-49af-935d-f33df1c6589b-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.655847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.655823    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"
Oct 31 13:36:26.674410 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.674385    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"]
Oct 31 13:36:26.680485 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.680058    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:36:26.681522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.681445    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:36:26.683008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.682772    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.683617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.683264    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab494491-9445-49af-935d-f33df1c6589b-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:26.703945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.699515    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-580f24-8t8pm\" (UID: \"b4d59eb6-d9aa-4795-8573-b08fb31d11b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"
Oct 31 13:36:26.703945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.699717    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwhb4\" (UniqueName: \"kubernetes.io/projected/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-kube-api-access-dwhb4\") pod \"maas-api-revoke-keys-e2e-api-a-580f24-8t8pm\" (UID: \"b4d59eb6-d9aa-4795-8573-b08fb31d11b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"
Oct 31 13:36:26.703945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.699799    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-580f24-8t8pm\" (UID: \"b4d59eb6-d9aa-4795-8573-b08fb31d11b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"
Oct 31 13:36:26.738104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.738045    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:26.789632 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.789597    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d18690c6-12a1-4bdf-a659-33d883ab758f" path="/var/lib/kubelet/pods/d18690c6-12a1-4bdf-a659-33d883ab758f/volumes"
Oct 31 13:36:26.800690 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.800655    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwhb4\" (UniqueName: \"kubernetes.io/projected/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-kube-api-access-dwhb4\") pod \"maas-api-revoke-keys-e2e-api-a-580f24-8t8pm\" (UID: \"b4d59eb6-d9aa-4795-8573-b08fb31d11b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"
Oct 31 13:36:26.800911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.800772    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-580f24-8t8pm\" (UID: \"b4d59eb6-d9aa-4795-8573-b08fb31d11b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"
Oct 31 13:36:26.800911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.800897    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-580f24-8t8pm\" (UID: \"b4d59eb6-d9aa-4795-8573-b08fb31d11b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"
Oct 31 13:36:26.802091 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.802064    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-580f24-8t8pm\" (UID: \"b4d59eb6-d9aa-4795-8573-b08fb31d11b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"
Oct 31 13:36:26.805013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.804988    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-580f24-8t8pm\" (UID: \"b4d59eb6-d9aa-4795-8573-b08fb31d11b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"
Oct 31 13:36:26.815148 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.815123    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwhb4\" (UniqueName: \"kubernetes.io/projected/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-kube-api-access-dwhb4\") pod \"maas-api-revoke-keys-e2e-api-a-580f24-8t8pm\" (UID: \"b4d59eb6-d9aa-4795-8573-b08fb31d11b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"
Oct 31 13:36:26.855943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.855833    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"
Oct 31 13:36:26.902419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.902384    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e34f0c7f-398d-4614-8851-913948829895-kube-api-access-qnrvq\" (UniqueName: \"kubernetes.io/projected/e34f0c7f-398d-4614-8851-913948829895-kube-api-access-qnrvq\") pod \"e34f0c7f-398d-4614-8851-913948829895\" (UID: \"e34f0c7f-398d-4614-8851-913948829895\") "
Oct 31 13:36:26.902589 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.902463    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e34f0c7f-398d-4614-8851-913948829895-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34f0c7f-398d-4614-8851-913948829895-plugins-config-volume\") pod \"e34f0c7f-398d-4614-8851-913948829895\" (UID: \"e34f0c7f-398d-4614-8851-913948829895\") "
Oct 31 13:36:26.902976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.902946    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34f0c7f-398d-4614-8851-913948829895-plugins-config-volume" pod "e34f0c7f-398d-4614-8851-913948829895" (UID: "e34f0c7f-398d-4614-8851-913948829895"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:26.903326 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.903301    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e34f0c7f-398d-4614-8851-913948829895-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:26.905372 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.905341    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34f0c7f-398d-4614-8851-913948829895-kube-api-access-qnrvq" pod "e34f0c7f-398d-4614-8851-913948829895" (UID: "e34f0c7f-398d-4614-8851-913948829895"). InnerVolumeSpecName "kube-api-access-qnrvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:26.947830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.947793    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:26.980343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:26.980306    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:27.005274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.005244    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnrvq\" (UniqueName: \"kubernetes.io/projected/e34f0c7f-398d-4614-8851-913948829895-kube-api-access-qnrvq\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:27.039529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.039383    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"
Oct 31 13:36:27.121435 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.121344    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:27.286882 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:36:27.286837    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab494491_9445_49af_935d_f33df1c6589b.slice/crio-34a8acb05c5ab5ee8fe42d1be1d2b286768f05f987bf37b12ad7d06038489775 WatchSource:0}: Error finding container 34a8acb05c5ab5ee8fe42d1be1d2b286768f05f987bf37b12ad7d06038489775: Status 404 returned error can't find the container with id 34a8acb05c5ab5ee8fe42d1be1d2b286768f05f987bf37b12ad7d06038489775
Oct 31 13:36:27.287773 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.287747    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"]
Oct 31 13:36:27.290172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.290139    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:27.290286 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.290200    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:27.290286 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.290229    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:27.308492 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.308460    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf" event={"ID":"ab494491-9445-49af-935d-f33df1c6589b","Type":"ContainerStarted","Data":"34a8acb05c5ab5ee8fe42d1be1d2b286768f05f987bf37b12ad7d06038489775"}
Oct 31 13:36:27.310049 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.310022    4428 generic.go:356] "Generic (PLEG): container finished" podID="e34f0c7f-398d-4614-8851-913948829895" containerID="1be9760b4fca4aa57c346eaae358b936f53cc5a718d7e515f41e2688cb861d1f" exitCode=0
Oct 31 13:36:27.310156 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.310084    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"
Oct 31 13:36:27.310156 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.310105    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9" event={"ID":"e34f0c7f-398d-4614-8851-913948829895","Type":"ContainerDied","Data":"1be9760b4fca4aa57c346eaae358b936f53cc5a718d7e515f41e2688cb861d1f"}
Oct 31 13:36:27.310285 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.310154    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9" event={"ID":"e34f0c7f-398d-4614-8851-913948829895","Type":"ContainerDied","Data":"0fbe19690f2f007208609e2601135ab287cb7341481be0c87c5f97e623e409e5"}
Oct 31 13:36:27.310285 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.310181    4428 scope.go:122] "RemoveContainer" containerID="1be9760b4fca4aa57c346eaae358b936f53cc5a718d7e515f41e2688cb861d1f"
Oct 31 13:36:27.341325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.341291    4428 scope.go:122] "RemoveContainer" containerID="1be9760b4fca4aa57c346eaae358b936f53cc5a718d7e515f41e2688cb861d1f"
Oct 31 13:36:27.341686 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:27.341654    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be9760b4fca4aa57c346eaae358b936f53cc5a718d7e515f41e2688cb861d1f\": container with ID starting with 1be9760b4fca4aa57c346eaae358b936f53cc5a718d7e515f41e2688cb861d1f not found: ID does not exist" containerID="1be9760b4fca4aa57c346eaae358b936f53cc5a718d7e515f41e2688cb861d1f"
Oct 31 13:36:27.341771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.341697    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be9760b4fca4aa57c346eaae358b936f53cc5a718d7e515f41e2688cb861d1f"} err="failed to get container status \"1be9760b4fca4aa57c346eaae358b936f53cc5a718d7e515f41e2688cb861d1f\": rpc error: code = NotFound desc = could not find container \"1be9760b4fca4aa57c346eaae358b936f53cc5a718d7e515f41e2688cb861d1f\": container with ID starting with 1be9760b4fca4aa57c346eaae358b936f53cc5a718d7e515f41e2688cb861d1f not found: ID does not exist"
Oct 31 13:36:27.363244 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.363215    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"]
Oct 31 13:36:27.368320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.368293    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cce3e7-86f5d98cb7-rcxn9"]
Oct 31 13:36:27.370954 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:36:27.370926    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d59eb6_d9aa_4795_8573_b08fb31d11b1.slice/crio-fcde5d1f72cc532b855b1ebc5c6a3063b0bf67c92a250e555608b991d480b2d5 WatchSource:0}: Error finding container fcde5d1f72cc532b855b1ebc5c6a3063b0bf67c92a250e555608b991d480b2d5: Status 404 returned error can't find the container with id fcde5d1f72cc532b855b1ebc5c6a3063b0bf67c92a250e555608b991d480b2d5
Oct 31 13:36:27.374094 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:27.374033    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"]
Oct 31 13:36:28.079147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.079117    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:28.184895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.182938    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"]
Oct 31 13:36:28.184895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.184155    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e34f0c7f-398d-4614-8851-913948829895" containerName="payload-pre-processing"
Oct 31 13:36:28.184895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.184179    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e34f0c7f-398d-4614-8851-913948829895" containerName="payload-pre-processing"
Oct 31 13:36:28.184895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.184445    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e34f0c7f-398d-4614-8851-913948829895" containerName="payload-pre-processing"
Oct 31 13:36:28.190336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.188697    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"]
Oct 31 13:36:28.190336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.188809    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"
Oct 31 13:36:28.193652 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.193620    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"]
Oct 31 13:36:28.197271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.197234    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-5e8f29\""
Oct 31 13:36:28.197401 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.197343    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-5e8f29-dockercfg-jqpnb\""
Oct 31 13:36:28.200980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.199642    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"
Oct 31 13:36:28.213845 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.213819    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"]
Oct 31 13:36:28.229326 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.229297    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6a83e5-80ac-4dca-bbce-fefc479e0abe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28\" (UID: \"ba6a83e5-80ac-4dca-bbce-fefc479e0abe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"
Oct 31 13:36:28.229592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.229573    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b042c863-4f10-4318-ac91-0464b31ebb89-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg\" (UID: \"b042c863-4f10-4318-ac91-0464b31ebb89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"
Oct 31 13:36:28.229772 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.229754    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpmn6\" (UniqueName: \"kubernetes.io/projected/ba6a83e5-80ac-4dca-bbce-fefc479e0abe-kube-api-access-bpmn6\") pod \"payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28\" (UID: \"ba6a83e5-80ac-4dca-bbce-fefc479e0abe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"
Oct 31 13:36:28.229903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.229881    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc9ql\" (UniqueName: \"kubernetes.io/projected/b042c863-4f10-4318-ac91-0464b31ebb89-kube-api-access-lc9ql\") pod \"payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg\" (UID: \"b042c863-4f10-4318-ac91-0464b31ebb89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"
Oct 31 13:36:28.260963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.248265    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:28.462209 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.462166    4428 generic.go:356] "Generic (PLEG): container finished" podID="50a90471-68bd-4059-ba8b-8719805293e8" containerID="ebaf9521a92268d7cd75b8bbaaf944dd891d705d2373851381286fec4dcb6e5d" exitCode=0
Oct 31 13:36:28.462989 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.462960    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt" event={"ID":"50a90471-68bd-4059-ba8b-8719805293e8","Type":"ContainerDied","Data":"ebaf9521a92268d7cd75b8bbaaf944dd891d705d2373851381286fec4dcb6e5d"}
Oct 31 13:36:28.499367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.488666    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6a83e5-80ac-4dca-bbce-fefc479e0abe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28\" (UID: \"ba6a83e5-80ac-4dca-bbce-fefc479e0abe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"
Oct 31 13:36:28.499367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.488989    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b042c863-4f10-4318-ac91-0464b31ebb89-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg\" (UID: \"b042c863-4f10-4318-ac91-0464b31ebb89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"
Oct 31 13:36:28.499367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.489172    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpmn6\" (UniqueName: \"kubernetes.io/projected/ba6a83e5-80ac-4dca-bbce-fefc479e0abe-kube-api-access-bpmn6\") pod \"payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28\" (UID: \"ba6a83e5-80ac-4dca-bbce-fefc479e0abe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"
Oct 31 13:36:28.499367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.489254    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc9ql\" (UniqueName: \"kubernetes.io/projected/b042c863-4f10-4318-ac91-0464b31ebb89-kube-api-access-lc9ql\") pod \"payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg\" (UID: \"b042c863-4f10-4318-ac91-0464b31ebb89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"
Oct 31 13:36:28.499367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.490423    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6a83e5-80ac-4dca-bbce-fefc479e0abe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28\" (UID: \"ba6a83e5-80ac-4dca-bbce-fefc479e0abe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"
Oct 31 13:36:28.499367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.491091    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b042c863-4f10-4318-ac91-0464b31ebb89-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg\" (UID: \"b042c863-4f10-4318-ac91-0464b31ebb89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"
Oct 31 13:36:28.507586 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.501630    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:28.520413 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.520386    4428 scope.go:122] "RemoveContainer" containerID="ebaf9521a92268d7cd75b8bbaaf944dd891d705d2373851381286fec4dcb6e5d"
Oct 31 13:36:28.562147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.561672    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc9ql\" (UniqueName: \"kubernetes.io/projected/b042c863-4f10-4318-ac91-0464b31ebb89-kube-api-access-lc9ql\") pod \"payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg\" (UID: \"b042c863-4f10-4318-ac91-0464b31ebb89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"
Oct 31 13:36:28.562661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.562638    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"
Oct 31 13:36:28.582917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.582077    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:28.582917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.582185    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:28.582917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.582239    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:28.667487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.663109    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpmn6\" (UniqueName: \"kubernetes.io/projected/ba6a83e5-80ac-4dca-bbce-fefc479e0abe-kube-api-access-bpmn6\") pod \"payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28\" (UID: \"ba6a83e5-80ac-4dca-bbce-fefc479e0abe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"
Oct 31 13:36:28.750835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.743568    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf" event={"ID":"ab494491-9445-49af-935d-f33df1c6589b","Type":"ContainerStarted","Data":"e99ee40f0c47df74e795c282abbaa510b8e995b9259a051b2b95a581f0e81741"}
Oct 31 13:36:28.785357 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.785316    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e34f0c7f-398d-4614-8851-913948829895" path="/var/lib/kubelet/pods/e34f0c7f-398d-4614-8851-913948829895/volumes"
Oct 31 13:36:28.788453 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.788417    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm" event={"ID":"b4d59eb6-d9aa-4795-8573-b08fb31d11b1","Type":"ContainerStarted","Data":"748a3e5a407aab8dfe2a69ab9c201372f648af409538b86ab96a639c3410ec6c"}
Oct 31 13:36:28.788617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.788476    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm" event={"ID":"b4d59eb6-d9aa-4795-8573-b08fb31d11b1","Type":"ContainerStarted","Data":"fcde5d1f72cc532b855b1ebc5c6a3063b0bf67c92a250e555608b991d480b2d5"}
Oct 31 13:36:28.790190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.790149    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf" podStartSLOduration=2.790136434 podStartE2EDuration="2.790136434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:28.782983485 +0000 UTC m=+1986.181332121" watchObservedRunningTime="2026-10-31 13:36:28.790136434 +0000 UTC m=+1986.188485115"
Oct 31 13:36:28.998746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.997453    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:28.998746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.997589    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 13:36:28.998746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:28.997632    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf" podUID="ab494491-9445-49af-935d-f33df1c6589b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 13:36:29.013816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:29.000073    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"
Oct 31 13:36:29.382349 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:36:29.382316    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb042c863_4f10_4318_ac91_0464b31ebb89.slice/crio-efc8fa0be62cba4aca63be10693fd00ab9962414fdbb3ac1891390af9f398555 WatchSource:0}: Error finding container efc8fa0be62cba4aca63be10693fd00ab9962414fdbb3ac1891390af9f398555: Status 404 returned error can't find the container with id efc8fa0be62cba4aca63be10693fd00ab9962414fdbb3ac1891390af9f398555
Oct 31 13:36:29.382464 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:29.382363    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"]
Oct 31 13:36:29.394061 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:29.391830    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"]
Oct 31 13:36:29.805016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:29.804114    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg" event={"ID":"b042c863-4f10-4318-ac91-0464b31ebb89","Type":"ContainerStarted","Data":"84ad66985cf7a6707d66b18f88efca9e49c2b28c522718d6a11eaebac3903cd4"}
Oct 31 13:36:29.805016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:29.804169    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg" event={"ID":"b042c863-4f10-4318-ac91-0464b31ebb89","Type":"ContainerStarted","Data":"efc8fa0be62cba4aca63be10693fd00ab9962414fdbb3ac1891390af9f398555"}
Oct 31 13:36:29.805016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:29.804372    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"
Oct 31 13:36:29.809610 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:29.809426    4428 generic.go:356] "Generic (PLEG): container finished" podID="b4d59eb6-d9aa-4795-8573-b08fb31d11b1" containerID="748a3e5a407aab8dfe2a69ab9c201372f648af409538b86ab96a639c3410ec6c" exitCode=0
Oct 31 13:36:29.809610 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:29.809516    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm" event={"ID":"b4d59eb6-d9aa-4795-8573-b08fb31d11b1","Type":"ContainerDied","Data":"748a3e5a407aab8dfe2a69ab9c201372f648af409538b86ab96a639c3410ec6c"}
Oct 31 13:36:29.817371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:29.814440    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28" event={"ID":"ba6a83e5-80ac-4dca-bbce-fefc479e0abe","Type":"ContainerStarted","Data":"3fb7763817dab9c9b09d1206e9a8b662fb121be571b2e77e3e22f579179097be"}
Oct 31 13:36:29.817371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:29.814510    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28" event={"ID":"ba6a83e5-80ac-4dca-bbce-fefc479e0abe","Type":"ContainerStarted","Data":"d941dcc6455f2d360fcf295345c1854853f512cfb93c8cbf836a996bdbc6f036"}
Oct 31 13:36:29.817371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:29.815983    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"
Oct 31 13:36:29.828925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:29.828881    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt" event={"ID":"50a90471-68bd-4059-ba8b-8719805293e8","Type":"ContainerStarted","Data":"5aa2ef4d9cfa75cdacb5ebe5f9137144e0c359afdf8a1073e0982001b94f07b5"}
Oct 31 13:36:29.870892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:29.859022    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg" podStartSLOduration=1.859002032 podStartE2EDuration="1.859002032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:29.843462488 +0000 UTC m=+1987.241811124" watchObservedRunningTime="2026-10-31 13:36:29.859002032 +0000 UTC m=+1987.257350657"
Oct 31 13:36:29.951593 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:29.951555    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:30.006890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:29.980231    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28" podStartSLOduration=1.980206881 podStartE2EDuration="1.980206881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:29.952534014 +0000 UTC m=+1987.350882639" watchObservedRunningTime="2026-10-31 13:36:29.980206881 +0000 UTC m=+1987.378555536"
Oct 31 13:36:30.019752 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:29.990646    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 13:36:30.019752 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:30.010057    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf" podUID="ab494491-9445-49af-935d-f33df1c6589b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 13:36:30.019752 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:29.990703    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:36:30.541291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:30.541265    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"
Oct 31 13:36:30.612159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:30.611678    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-maas-api-xks-ca\") pod \"b4d59eb6-d9aa-4795-8573-b08fb31d11b1\" (UID: \"b4d59eb6-d9aa-4795-8573-b08fb31d11b1\") "
Oct 31 13:36:30.612159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:30.611804    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-maas-api-openshift-ca\") pod \"b4d59eb6-d9aa-4795-8573-b08fb31d11b1\" (UID: \"b4d59eb6-d9aa-4795-8573-b08fb31d11b1\") "
Oct 31 13:36:30.612159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:30.611892    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-kube-api-access-dwhb4\" (UniqueName: \"kubernetes.io/projected/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-kube-api-access-dwhb4\") pod \"b4d59eb6-d9aa-4795-8573-b08fb31d11b1\" (UID: \"b4d59eb6-d9aa-4795-8573-b08fb31d11b1\") "
Oct 31 13:36:30.613347 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:30.613318    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-maas-api-openshift-ca" pod "b4d59eb6-d9aa-4795-8573-b08fb31d11b1" (UID: "b4d59eb6-d9aa-4795-8573-b08fb31d11b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:30.615542 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:30.615510    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-maas-api-xks-ca" pod "b4d59eb6-d9aa-4795-8573-b08fb31d11b1" (UID: "b4d59eb6-d9aa-4795-8573-b08fb31d11b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:36:30.617041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:30.617015    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-kube-api-access-dwhb4" pod "b4d59eb6-d9aa-4795-8573-b08fb31d11b1" (UID: "b4d59eb6-d9aa-4795-8573-b08fb31d11b1"). InnerVolumeSpecName "kube-api-access-dwhb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:30.714781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:30.714353    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:30.714781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:30.714391    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:30.714781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:30.714407    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwhb4\" (UniqueName: \"kubernetes.io/projected/b4d59eb6-d9aa-4795-8573-b08fb31d11b1-kube-api-access-dwhb4\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:30.837554 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:30.837525    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"
Oct 31 13:36:30.838231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:30.838203    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm" event={"ID":"b4d59eb6-d9aa-4795-8573-b08fb31d11b1","Type":"ContainerDied","Data":"fcde5d1f72cc532b855b1ebc5c6a3063b0bf67c92a250e555608b991d480b2d5"}
Oct 31 13:36:30.838375 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:30.838261    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcde5d1f72cc532b855b1ebc5c6a3063b0bf67c92a250e555608b991d480b2d5"
Oct 31 13:36:30.941927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:30.937676    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:30.992946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:30.983131    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 13:36:30.992946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:30.983195    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf" podUID="ab494491-9445-49af-935d-f33df1c6589b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 13:36:31.080592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:31.080541    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 13:36:31.088958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:31.088843    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt" podUID="50a90471-68bd-4059-ba8b-8719805293e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 13:36:31.973070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:31.970399    4428 scope.go:122] "RemoveContainer" containerID="c8697012c373505430b03a00d9c0d819f863bbb2acde265c84362ae88d9523f2"
Oct 31 13:36:31.988027 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:31.986584    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 13:36:31.988027 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:31.986638    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf" podUID="ab494491-9445-49af-935d-f33df1c6589b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 13:36:31.988611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:31.988402    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 13:36:31.988611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:31.988438    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt" podUID="50a90471-68bd-4059-ba8b-8719805293e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 13:36:32.014092 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:32.014068    4428 scope.go:122] "RemoveContainer" containerID="93edfad28f095df4e51d643de483067490c29b8ac6d42ec7d4eba0f6b3a5cddf"
Oct 31 13:36:32.104895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:32.087105    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:32.143140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:32.121752    4428 scope.go:122] "RemoveContainer" containerID="7ac0b0b9a53e2db599852369b48cbd8da05a48290f875b4cd8c68ffba172bc92"
Oct 31 13:36:32.232985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:32.216317    4428 scope.go:122] "RemoveContainer" containerID="f1fa4c7b66c91074561341ae7ed6b9d14224005687862750676408e91e65f31a"
Oct 31 13:36:32.305710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:32.305679    4428 scope.go:122] "RemoveContainer" containerID="9e22c386611637a30c14a93f8b41095416afb88dab31b1328046529aeb45f69f"
Oct 31 13:36:32.317474 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:32.317453    4428 scope.go:122] "RemoveContainer" containerID="63dc0e408c496a7661acfcae49b6560e4edb77e9f161961568314c2ede6120cf"
Oct 31 13:36:32.904952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:32.902171    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:33.004892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.001590    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 13:36:33.004892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.001654    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf" podUID="ab494491-9445-49af-935d-f33df1c6589b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 13:36:33.004892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.002098    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 13:36:33.004892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.002148    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt" podUID="50a90471-68bd-4059-ba8b-8719805293e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 13:36:33.843052 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.841553    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"]
Oct 31 13:36:33.843052 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.842908    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b4d59eb6-d9aa-4795-8573-b08fb31d11b1" containerName="revoke-keys"
Oct 31 13:36:33.843052 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.842926    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d59eb6-d9aa-4795-8573-b08fb31d11b1" containerName="revoke-keys"
Oct 31 13:36:33.843266 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.843066    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d59eb6-d9aa-4795-8573-b08fb31d11b1" containerName="revoke-keys"
Oct 31 13:36:33.850679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.850636    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:36:33.905699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.905200    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c45a37-serving-cert\""
Oct 31 13:36:33.911360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.909512    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-maas-api-tls\") pod \"maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66\" (UID: \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:36:33.911360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.909706    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c64vc\" (UniqueName: \"kubernetes.io/projected/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-kube-api-access-c64vc\") pod \"maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66\" (UID: \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:36:33.911360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.909757    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-openshift-service-ca\") pod \"maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66\" (UID: \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:36:33.911360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.910030    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-metrics-certs\") pod \"maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66\" (UID: \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:36:33.913731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.913278    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"]
Oct 31 13:36:34.023986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.986325    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:34.023986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.986497    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 13:36:34.023986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:33.986544    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt" podUID="50a90471-68bd-4059-ba8b-8719805293e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 13:36:34.023986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.014427    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-openshift-service-ca\") pod \"maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66\" (UID: \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:36:34.023986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.014700    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-metrics-certs\") pod \"maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66\" (UID: \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:36:34.023986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.014920    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-maas-api-tls\") pod \"maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66\" (UID: \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:36:34.023986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.015012    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c64vc\" (UniqueName: \"kubernetes.io/projected/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-kube-api-access-c64vc\") pod \"maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66\" (UID: \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:36:34.049902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.026611    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-openshift-service-ca\") pod \"maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66\" (UID: \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:36:34.049902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.028326    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 13:36:34.049902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.028375    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf" podUID="ab494491-9445-49af-935d-f33df1c6589b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 13:36:34.194935 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.152132    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:34.211045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.155008    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c64vc\" (UniqueName: \"kubernetes.io/projected/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-kube-api-access-c64vc\") pod \"maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66\" (UID: \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:36:34.234983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.194449    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-maas-api-tls\") pod \"maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66\" (UID: \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:36:34.235250 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.214459    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-metrics-certs\") pod \"maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66\" (UID: \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:36:34.558891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.532376    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:36:34.664488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.633330    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:34.664488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.651420    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"]
Oct 31 13:36:34.664488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.651664    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw" podUID="d3179e22-1592-44a5-8193-8e7e9aee4b5e" containerName="istio-proxy" containerID="cri-o://dae3cc317ff5a8055e68f2c6b703a6020bf92eb12800d0f5f9e4cd35b4f4d72e" gracePeriod=30
Oct 31 13:36:34.996188 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.996151    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 13:36:34.996480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.996446    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt" podUID="50a90471-68bd-4059-ba8b-8719805293e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 13:36:34.997299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:34.997274    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 13:36:35.031933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:35.031891    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf" podUID="ab494491-9445-49af-935d-f33df1c6589b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 13:36:35.038985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:35.034942    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:36:35.231887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:35.231822    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:35.287008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:35.276085    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:36:35.437023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:35.436643    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:36:35.508698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:35.504007    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Oct 31 13:36:35.508698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:35.504289    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:36:35.587944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:35.540503    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:35.608909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:35.607631    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"]
Oct 31 13:36:35.622889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:35.615426    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:35.627825 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:35.627309    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"]
Oct 31 13:36:35.627825 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:35.627649    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-69bdf5-openshift-default-dockercfg-v2qtd\""
Oct 31 13:36:35.723595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:35.723562    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:36.039425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.017763    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x67c2\" (UniqueName: \"kubernetes.io/projected/c080a436-0943-4c33-938c-35b1ff2c1a11-kube-api-access-x67c2\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.039425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.017959    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-workload-certs\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.039425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.018062    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-workload-socket\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.039425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.018238    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-data\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.039425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.018294    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-envoy\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.039425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.018332    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c080a436-0943-4c33-938c-35b1ff2c1a11-istiod-ca-cert\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.039425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.018463    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-credential-socket\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.039425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.018511    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-podinfo\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.039425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.018561    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-token\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.039425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.030372    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 13:36:36.039425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.030426    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt" podUID="50a90471-68bd-4059-ba8b-8719805293e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 13:36:36.071825 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.061992    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:36.071825 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.070465    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"]
Oct 31 13:36:36.083361 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.078229    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:36.083361 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.080002    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:36.135681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.135649    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:36:36.138528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.138457    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-credential-socket\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.140987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.140958    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-podinfo\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.143909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.143824    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-token\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.144182 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.140321    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-credential-socket\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.144639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.144225    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x67c2\" (UniqueName: \"kubernetes.io/projected/c080a436-0943-4c33-938c-35b1ff2c1a11-kube-api-access-x67c2\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.144963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.144945    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-workload-certs\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.145187 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.145168    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-workload-socket\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.145456 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.145439    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-data\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.145596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.145582    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-envoy\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.145714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.145700    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c080a436-0943-4c33-938c-35b1ff2c1a11-istiod-ca-cert\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.173902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.146100    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-workload-socket\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.173902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.146342    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-workload-certs\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.173902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.146823    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-data\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.175109 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.175073    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c080a436-0943-4c33-938c-35b1ff2c1a11-istiod-ca-cert\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.181894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.179826    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-podinfo\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.193083 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.191654    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66" event={"ID":"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2","Type":"ContainerStarted","Data":"2823dca9b71de988235fd4f1057ac47e212b386dbdded1e6f5203d1b30e89b9c"}
Oct 31 13:36:36.220761 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.220723    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-token\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.266544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.224152    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x67c2\" (UniqueName: \"kubernetes.io/projected/c080a436-0943-4c33-938c-35b1ff2c1a11-kube-api-access-x67c2\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.266544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.254946    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-envoy\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.327783 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.327225    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:36.679942 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:36:36.679913    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc080a436_0943_4c33_938c_35b1ff2c1a11.slice/crio-9b92fb041dd2cdd9aa7afbff08cfde337fbb0c3d5c23124ce5b07bf80d64cb05 WatchSource:0}: Error finding container 9b92fb041dd2cdd9aa7afbff08cfde337fbb0c3d5c23124ce5b07bf80d64cb05: Status 404 returned error can't find the container with id 9b92fb041dd2cdd9aa7afbff08cfde337fbb0c3d5c23124ce5b07bf80d64cb05
Oct 31 13:36:36.680778 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.680751    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"]
Oct 31 13:36:36.683065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.683030    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:36.683139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.683100    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:36.683139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.683129    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:36.746233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.746203    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"]
Oct 31 13:36:36.877005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.871425    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:36.991707 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.991671    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:36:36.993040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:36.993015    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:36:37.058897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.046606    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"]
Oct 31 13:36:37.079655 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.079492    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:37.093675 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.093649    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.202415 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.201563    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"]
Oct 31 13:36:37.326997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.282834    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.326997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.282908    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.326997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.282984    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.326997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.283039    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.326997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.283165    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.326997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.283194    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drspc\" (UniqueName: \"kubernetes.io/projected/65d39130-1203-4a53-af59-cd0d7c2afd57-kube-api-access-drspc\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.326997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.283250    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.326997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.283374    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.326997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.283537    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d39130-1203-4a53-af59-cd0d7c2afd57-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.395974 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.387024    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d39130-1203-4a53-af59-cd0d7c2afd57-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.396619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.396590    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.498815 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.498781    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.500662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.500465    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.528422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.388720    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d39130-1203-4a53-af59-cd0d7c2afd57-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.528422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.407421    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.528422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.510069    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.529289 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.529253    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.529548 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.529501    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.529630 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.529577    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drspc\" (UniqueName: \"kubernetes.io/projected/65d39130-1203-4a53-af59-cd0d7c2afd57-kube-api-access-drspc\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.529687 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.529648    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.529811 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.529790    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.537891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.536006    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.541623 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.541343    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.545816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.544996    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.555914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.555350    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66" event={"ID":"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2","Type":"ContainerStarted","Data":"e2eb6288b90f0620b7bf829340c7475019d26104cdb8a12d15943470ca78b91c"}
Oct 31 13:36:37.557211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.556116    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:36:37.558353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.558166    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.561710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.560583    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drspc\" (UniqueName: \"kubernetes.io/projected/65d39130-1203-4a53-af59-cd0d7c2afd57-kube-api-access-drspc\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.564507 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.563782    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x" event={"ID":"c080a436-0943-4c33-938c-35b1ff2c1a11","Type":"ContainerStarted","Data":"d6ebff4bee6e489501f3c38883f9f91780938f8f20ade5c84b5c7d5ac26085a8"}
Oct 31 13:36:37.564507 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.563831    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x" event={"ID":"c080a436-0943-4c33-938c-35b1ff2c1a11","Type":"ContainerStarted","Data":"9b92fb041dd2cdd9aa7afbff08cfde337fbb0c3d5c23124ce5b07bf80d64cb05"}
Oct 31 13:36:37.565088 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.565063    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:37.593635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.593550    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66" podStartSLOduration=4.030219328 podStartE2EDuration="4.593537456s" totalImagesPullingTime="563.318128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:37.590929936 +0000 UTC m=+1994.989278559" watchObservedRunningTime="2026-10-31 13:36:37.593537456 +0000 UTC m=+1994.991886079"
Oct 31 13:36:37.620072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.620012    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x" podStartSLOduration=2.6199992290000003 podStartE2EDuration="2.619999229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:37.616914008 +0000 UTC m=+1995.015262634" watchObservedRunningTime="2026-10-31 13:36:37.619999229 +0000 UTC m=+1995.018347852"
Oct 31 13:36:37.861458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:37.861041    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:38.162144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:38.162118    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"]
Oct 31 13:36:38.164652 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:38.164614    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:38.164741 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:38.164700    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:38.164797 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:38.164741    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:38.249116 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:38.244406    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:38.340926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:38.334184    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:38.353324 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:38.353187    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:38.575096 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:38.575013    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb" event={"ID":"65d39130-1203-4a53-af59-cd0d7c2afd57","Type":"ContainerStarted","Data":"4a085768ec72537a1fc02bbf9aef27a06b48d3bd50af84d8625d8fb4a0517c80"}
Oct 31 13:36:38.575096 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:38.575082    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb" event={"ID":"65d39130-1203-4a53-af59-cd0d7c2afd57","Type":"ContainerStarted","Data":"6774d8c64da365beea86299f935fb4b6f3d34a79e93ccfa2e727fcfc0f66bacf"}
Oct 31 13:36:38.575916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:38.575859    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:38.576802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:38.576783    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:36:38.601762 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:38.601708    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb" podStartSLOduration=1.601691644 podStartE2EDuration="1.601691644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:38.600236715 +0000 UTC m=+1995.998585374" watchObservedRunningTime="2026-10-31 13:36:38.601691644 +0000 UTC m=+1996.000040269"
Oct 31 13:36:38.864954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:38.864033    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:39.612724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:39.612693    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:39.875306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:39.874923    4428 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 31 13:36:39.875306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:39.874984    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb" podUID="65d39130-1203-4a53-af59-cd0d7c2afd57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 31 13:36:40.006351 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.006309    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:40.116419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.116395    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:36:40.177792 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.177719    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:40.226475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.226446    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-envoy\") pod \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") "
Oct 31 13:36:40.226656 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.226492    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istiod-ca-cert\") pod \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") "
Oct 31 13:36:40.226656 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.226633    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3179e22-1592-44a5-8193-8e7e9aee4b5e-kube-api-access-rtvv8\" (UniqueName: \"kubernetes.io/projected/d3179e22-1592-44a5-8193-8e7e9aee4b5e-kube-api-access-rtvv8\") pod \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") "
Oct 31 13:36:40.226771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.226714    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-token\" (UniqueName: \"kubernetes.io/projected/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-token\") pod \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") "
Oct 31 13:36:40.226771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.226753    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-workload-socket\") pod \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") "
Oct 31 13:36:40.226901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.226793    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-data\") pod \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") "
Oct 31 13:36:40.226901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.226835    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-workload-certs\") pod \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") "
Oct 31 13:36:40.227011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.226905    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-credential-socket\") pod \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") "
Oct 31 13:36:40.227011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.226957    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-podinfo\") pod \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\" (UID: \"d3179e22-1592-44a5-8193-8e7e9aee4b5e\") "
Oct 31 13:36:40.227136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.226999    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istiod-ca-cert" pod "d3179e22-1592-44a5-8193-8e7e9aee4b5e" (UID: "d3179e22-1592-44a5-8193-8e7e9aee4b5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:40.227287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.227265    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-data" pod "d3179e22-1592-44a5-8193-8e7e9aee4b5e" (UID: "d3179e22-1592-44a5-8193-8e7e9aee4b5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:40.227470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.227447    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-workload-socket" pod "d3179e22-1592-44a5-8193-8e7e9aee4b5e" (UID: "d3179e22-1592-44a5-8193-8e7e9aee4b5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:40.227977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.227951    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:40.228101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.227981    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:40.228101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.227995    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:40.228496 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.228465    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-credential-socket" pod "d3179e22-1592-44a5-8193-8e7e9aee4b5e" (UID: "d3179e22-1592-44a5-8193-8e7e9aee4b5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:40.229029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.228998    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-workload-certs" pod "d3179e22-1592-44a5-8193-8e7e9aee4b5e" (UID: "d3179e22-1592-44a5-8193-8e7e9aee4b5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:40.229596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.229564    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-envoy" pod "d3179e22-1592-44a5-8193-8e7e9aee4b5e" (UID: "d3179e22-1592-44a5-8193-8e7e9aee4b5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:40.229766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.229744    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-token" pod "d3179e22-1592-44a5-8193-8e7e9aee4b5e" (UID: "d3179e22-1592-44a5-8193-8e7e9aee4b5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:40.229943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.229922    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-podinfo" pod "d3179e22-1592-44a5-8193-8e7e9aee4b5e" (UID: "d3179e22-1592-44a5-8193-8e7e9aee4b5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:36:40.230730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.230705    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3179e22-1592-44a5-8193-8e7e9aee4b5e-kube-api-access-rtvv8" pod "d3179e22-1592-44a5-8193-8e7e9aee4b5e" (UID: "d3179e22-1592-44a5-8193-8e7e9aee4b5e"). InnerVolumeSpecName "kube-api-access-rtvv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:40.330004 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.329974    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:40.330004 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.330001    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtvv8\" (UniqueName: \"kubernetes.io/projected/d3179e22-1592-44a5-8193-8e7e9aee4b5e-kube-api-access-rtvv8\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:40.330004 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.330011    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:40.330238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.330020    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:40.330238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.330038    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3179e22-1592-44a5-8193-8e7e9aee4b5e-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:40.330238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.330051    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3179e22-1592-44a5-8193-8e7e9aee4b5e-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:40.598449 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.598411    4428 generic.go:356] "Generic (PLEG): container finished" podID="d3179e22-1592-44a5-8193-8e7e9aee4b5e" containerID="dae3cc317ff5a8055e68f2c6b703a6020bf92eb12800d0f5f9e4cd35b4f4d72e" exitCode=0
Oct 31 13:36:40.598660 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.598485    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"
Oct 31 13:36:40.598660 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.598505    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw" event={"ID":"d3179e22-1592-44a5-8193-8e7e9aee4b5e","Type":"ContainerDied","Data":"dae3cc317ff5a8055e68f2c6b703a6020bf92eb12800d0f5f9e4cd35b4f4d72e"}
Oct 31 13:36:40.598660 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.598610    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw" event={"ID":"d3179e22-1592-44a5-8193-8e7e9aee4b5e","Type":"ContainerDied","Data":"82056bf2fd0e5acb98d23da210a4bf6523c8380b31ae07813efda9f923c97557"}
Oct 31 13:36:40.598660 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.598645    4428 scope.go:122] "RemoveContainer" containerID="dae3cc317ff5a8055e68f2c6b703a6020bf92eb12800d0f5f9e4cd35b4f4d72e"
Oct 31 13:36:40.612141 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.612126    4428 scope.go:122] "RemoveContainer" containerID="cbb534617b29013f4c84e26ed1b7c2bf413c252d966860f002d75069f26b5e10"
Oct 31 13:36:40.622460 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.622438    4428 scope.go:122] "RemoveContainer" containerID="dae3cc317ff5a8055e68f2c6b703a6020bf92eb12800d0f5f9e4cd35b4f4d72e"
Oct 31 13:36:40.622819 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:40.622741    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae3cc317ff5a8055e68f2c6b703a6020bf92eb12800d0f5f9e4cd35b4f4d72e\": container with ID starting with dae3cc317ff5a8055e68f2c6b703a6020bf92eb12800d0f5f9e4cd35b4f4d72e not found: ID does not exist" containerID="dae3cc317ff5a8055e68f2c6b703a6020bf92eb12800d0f5f9e4cd35b4f4d72e"
Oct 31 13:36:40.622819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.622790    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae3cc317ff5a8055e68f2c6b703a6020bf92eb12800d0f5f9e4cd35b4f4d72e"} err="failed to get container status \"dae3cc317ff5a8055e68f2c6b703a6020bf92eb12800d0f5f9e4cd35b4f4d72e\": rpc error: code = NotFound desc = could not find container \"dae3cc317ff5a8055e68f2c6b703a6020bf92eb12800d0f5f9e4cd35b4f4d72e\": container with ID starting with dae3cc317ff5a8055e68f2c6b703a6020bf92eb12800d0f5f9e4cd35b4f4d72e not found: ID does not exist"
Oct 31 13:36:40.622819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.622812    4428 scope.go:122] "RemoveContainer" containerID="cbb534617b29013f4c84e26ed1b7c2bf413c252d966860f002d75069f26b5e10"
Oct 31 13:36:40.623211 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:40.623165    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb534617b29013f4c84e26ed1b7c2bf413c252d966860f002d75069f26b5e10\": container with ID starting with cbb534617b29013f4c84e26ed1b7c2bf413c252d966860f002d75069f26b5e10 not found: ID does not exist" containerID="cbb534617b29013f4c84e26ed1b7c2bf413c252d966860f002d75069f26b5e10"
Oct 31 13:36:40.623330 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.623220    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb534617b29013f4c84e26ed1b7c2bf413c252d966860f002d75069f26b5e10"} err="failed to get container status \"cbb534617b29013f4c84e26ed1b7c2bf413c252d966860f002d75069f26b5e10\": rpc error: code = NotFound desc = could not find container \"cbb534617b29013f4c84e26ed1b7c2bf413c252d966860f002d75069f26b5e10\": container with ID starting with cbb534617b29013f4c84e26ed1b7c2bf413c252d966860f002d75069f26b5e10 not found: ID does not exist"
Oct 31 13:36:40.631312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.631292    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"]
Oct 31 13:36:40.635979 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.635948    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-752869-openshift-default-9cf7bfd89-r49sw"]
Oct 31 13:36:40.780187 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.780156    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3179e22-1592-44a5-8193-8e7e9aee4b5e" path="/var/lib/kubelet/pods/d3179e22-1592-44a5-8193-8e7e9aee4b5e/volumes"
Oct 31 13:36:40.842058 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.842036    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"
Oct 31 13:36:40.867941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.866402    4428 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 31 13:36:40.867941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.866477    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb" podUID="65d39130-1203-4a53-af59-cd0d7c2afd57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 31 13:36:40.991650 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:40.991558    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:41.409813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.409785    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:41.433967 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.433939    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"]
Oct 31 13:36:41.434667 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.434653    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d3179e22-1592-44a5-8193-8e7e9aee4b5e" containerName="istio-proxy"
Oct 31 13:36:41.434718 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.434669    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3179e22-1592-44a5-8193-8e7e9aee4b5e" containerName="istio-proxy"
Oct 31 13:36:41.434806 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.434795    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3179e22-1592-44a5-8193-8e7e9aee4b5e" containerName="istio-proxy"
Oct 31 13:36:41.434851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.434809    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3179e22-1592-44a5-8193-8e7e9aee4b5e" containerName="istio-proxy"
Oct 31 13:36:41.440478 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.440457    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"
Oct 31 13:36:41.454060 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.454015    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-69bdf5-dockercfg-g7gkq\""
Oct 31 13:36:41.484783 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.484748    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"]
Oct 31 13:36:41.498778 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.498224    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"]
Oct 31 13:36:41.499317 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.499163    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d3179e22-1592-44a5-8193-8e7e9aee4b5e" containerName="istio-proxy"
Oct 31 13:36:41.499317 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.499194    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3179e22-1592-44a5-8193-8e7e9aee4b5e" containerName="istio-proxy"
Oct 31 13:36:41.503929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.503661    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"
Oct 31 13:36:41.532634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.519480    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"]
Oct 31 13:36:41.532634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.523473    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-69bdf5\""
Oct 31 13:36:41.676940 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.667776    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:41.805896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.804542    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9" podUID="767378d1-3634-4db5-873f-7cc9fc3fb5a9" containerName="payload-pre-processing" containerID="cri-o://6c5891d5958600fec530f7ae5e814a03f3e3f88aa36dd3f143a3a31ebfd14723" gracePeriod=25
Oct 31 13:36:41.889913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.887255    4428 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 31 13:36:41.889913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.887331    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb" podUID="65d39130-1203-4a53-af59-cd0d7c2afd57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 31 13:36:41.956516 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.950306    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcrr4\" (UniqueName: \"kubernetes.io/projected/371c7f96-669c-455b-a063-29303ee30cb2-kube-api-access-pcrr4\") pod \"payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s\" (UID: \"371c7f96-669c-455b-a063-29303ee30cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"
Oct 31 13:36:41.956516 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.950383    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371c7f96-669c-455b-a063-29303ee30cb2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s\" (UID: \"371c7f96-669c-455b-a063-29303ee30cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"
Oct 31 13:36:41.972895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.967087    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj7kp\" (UniqueName: \"kubernetes.io/projected/b83d1055-13eb-489e-9bc2-17780894bcfb-kube-api-access-fj7kp\") pod \"payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d\" (UID: \"b83d1055-13eb-489e-9bc2-17780894bcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"
Oct 31 13:36:42.039982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.005938    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83d1055-13eb-489e-9bc2-17780894bcfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d\" (UID: \"b83d1055-13eb-489e-9bc2-17780894bcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"
Oct 31 13:36:42.039982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:41.990718    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"
Oct 31 13:36:42.108192 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.108150    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83d1055-13eb-489e-9bc2-17780894bcfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d\" (UID: \"b83d1055-13eb-489e-9bc2-17780894bcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"
Oct 31 13:36:42.108446 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.108423    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcrr4\" (UniqueName: \"kubernetes.io/projected/371c7f96-669c-455b-a063-29303ee30cb2-kube-api-access-pcrr4\") pod \"payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s\" (UID: \"371c7f96-669c-455b-a063-29303ee30cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"
Oct 31 13:36:42.108535 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.108469    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371c7f96-669c-455b-a063-29303ee30cb2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s\" (UID: \"371c7f96-669c-455b-a063-29303ee30cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"
Oct 31 13:36:42.108620 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.108569    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj7kp\" (UniqueName: \"kubernetes.io/projected/b83d1055-13eb-489e-9bc2-17780894bcfb-kube-api-access-fj7kp\") pod \"payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d\" (UID: \"b83d1055-13eb-489e-9bc2-17780894bcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"
Oct 31 13:36:42.109062 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.109032    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83d1055-13eb-489e-9bc2-17780894bcfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d\" (UID: \"b83d1055-13eb-489e-9bc2-17780894bcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"
Oct 31 13:36:42.109630 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.109608    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371c7f96-669c-455b-a063-29303ee30cb2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s\" (UID: \"371c7f96-669c-455b-a063-29303ee30cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"
Oct 31 13:36:42.124154 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.124128    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcrr4\" (UniqueName: \"kubernetes.io/projected/371c7f96-669c-455b-a063-29303ee30cb2-kube-api-access-pcrr4\") pod \"payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s\" (UID: \"371c7f96-669c-455b-a063-29303ee30cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"
Oct 31 13:36:42.124664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.124642    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj7kp\" (UniqueName: \"kubernetes.io/projected/b83d1055-13eb-489e-9bc2-17780894bcfb-kube-api-access-fj7kp\") pod \"payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d\" (UID: \"b83d1055-13eb-489e-9bc2-17780894bcfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"
Oct 31 13:36:42.129605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.129577    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"
Oct 31 13:36:42.391396 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.391362    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"
Oct 31 13:36:42.473895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.473199    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:42.695542 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.680224    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"]
Oct 31 13:36:42.741191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.741157    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"
Oct 31 13:36:42.817900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.816620    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/767378d1-3634-4db5-873f-7cc9fc3fb5a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767378d1-3634-4db5-873f-7cc9fc3fb5a9-plugins-config-volume\") pod \"767378d1-3634-4db5-873f-7cc9fc3fb5a9\" (UID: \"767378d1-3634-4db5-873f-7cc9fc3fb5a9\") "
Oct 31 13:36:42.817900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.816915    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767378d1-3634-4db5-873f-7cc9fc3fb5a9-kube-api-access-ndq7c\" (UniqueName: \"kubernetes.io/projected/767378d1-3634-4db5-873f-7cc9fc3fb5a9-kube-api-access-ndq7c\") pod \"767378d1-3634-4db5-873f-7cc9fc3fb5a9\" (UID: \"767378d1-3634-4db5-873f-7cc9fc3fb5a9\") "
Oct 31 13:36:42.817900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.817401    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767378d1-3634-4db5-873f-7cc9fc3fb5a9-plugins-config-volume" pod "767378d1-3634-4db5-873f-7cc9fc3fb5a9" (UID: "767378d1-3634-4db5-873f-7cc9fc3fb5a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:42.818169 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.818095    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767378d1-3634-4db5-873f-7cc9fc3fb5a9-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:42.829091 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.828357    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767378d1-3634-4db5-873f-7cc9fc3fb5a9-kube-api-access-ndq7c" pod "767378d1-3634-4db5-873f-7cc9fc3fb5a9" (UID: "767378d1-3634-4db5-873f-7cc9fc3fb5a9"). InnerVolumeSpecName "kube-api-access-ndq7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:42.858608 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.858574    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"]
Oct 31 13:36:42.869674 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.869644    4428 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 31 13:36:42.869800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.869691    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb" podUID="65d39130-1203-4a53-af59-cd0d7c2afd57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 31 13:36:42.887592 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:36:42.887514    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83d1055_13eb_489e_9bc2_17780894bcfb.slice/crio-e6bf00259f76b1cf4a2389df3ad19a54c6780059847fc989eea4e8d57ce9502f WatchSource:0}: Error finding container e6bf00259f76b1cf4a2389df3ad19a54c6780059847fc989eea4e8d57ce9502f: Status 404 returned error can't find the container with id e6bf00259f76b1cf4a2389df3ad19a54c6780059847fc989eea4e8d57ce9502f
Oct 31 13:36:42.919730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.919708    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndq7c\" (UniqueName: \"kubernetes.io/projected/767378d1-3634-4db5-873f-7cc9fc3fb5a9-kube-api-access-ndq7c\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:42.987170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.987138    4428 generic.go:356] "Generic (PLEG): container finished" podID="767378d1-3634-4db5-873f-7cc9fc3fb5a9" containerID="6c5891d5958600fec530f7ae5e814a03f3e3f88aa36dd3f143a3a31ebfd14723" exitCode=0
Oct 31 13:36:42.987306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.987191    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"
Oct 31 13:36:42.987306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.987261    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9" event={"ID":"767378d1-3634-4db5-873f-7cc9fc3fb5a9","Type":"ContainerDied","Data":"6c5891d5958600fec530f7ae5e814a03f3e3f88aa36dd3f143a3a31ebfd14723"}
Oct 31 13:36:42.987410 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.987313    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9" event={"ID":"767378d1-3634-4db5-873f-7cc9fc3fb5a9","Type":"ContainerDied","Data":"ff67fc16dc9ce75c65f4758e7d38868198a894160aba50ac404b2d280fe5d4aa"}
Oct 31 13:36:42.987410 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.987341    4428 scope.go:122] "RemoveContainer" containerID="6c5891d5958600fec530f7ae5e814a03f3e3f88aa36dd3f143a3a31ebfd14723"
Oct 31 13:36:42.990475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.989714    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d" event={"ID":"b83d1055-13eb-489e-9bc2-17780894bcfb","Type":"ContainerStarted","Data":"e6bf00259f76b1cf4a2389df3ad19a54c6780059847fc989eea4e8d57ce9502f"}
Oct 31 13:36:42.990475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.990236    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"
Oct 31 13:36:42.992438 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.992411    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s" event={"ID":"371c7f96-669c-455b-a063-29303ee30cb2","Type":"ContainerStarted","Data":"9999772c1ecb24b4f956e4cb881e244522290f2edc77289f0f17888bd6868b84"}
Oct 31 13:36:42.992562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.992454    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s" event={"ID":"371c7f96-669c-455b-a063-29303ee30cb2","Type":"ContainerStarted","Data":"7b224fca0a6c95881d4c72788ccdcc8f608e6b5de07831d888ee7d0a66d72809"}
Oct 31 13:36:42.992562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.992556    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"
Oct 31 13:36:42.998886 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.998851    4428 scope.go:122] "RemoveContainer" containerID="6c5891d5958600fec530f7ae5e814a03f3e3f88aa36dd3f143a3a31ebfd14723"
Oct 31 13:36:42.999147 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:42.999126    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5891d5958600fec530f7ae5e814a03f3e3f88aa36dd3f143a3a31ebfd14723\": container with ID starting with 6c5891d5958600fec530f7ae5e814a03f3e3f88aa36dd3f143a3a31ebfd14723 not found: ID does not exist" containerID="6c5891d5958600fec530f7ae5e814a03f3e3f88aa36dd3f143a3a31ebfd14723"
Oct 31 13:36:42.999222 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:42.999159    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5891d5958600fec530f7ae5e814a03f3e3f88aa36dd3f143a3a31ebfd14723"} err="failed to get container status \"6c5891d5958600fec530f7ae5e814a03f3e3f88aa36dd3f143a3a31ebfd14723\": rpc error: code = NotFound desc = could not find container \"6c5891d5958600fec530f7ae5e814a03f3e3f88aa36dd3f143a3a31ebfd14723\": container with ID starting with 6c5891d5958600fec530f7ae5e814a03f3e3f88aa36dd3f143a3a31ebfd14723 not found: ID does not exist"
Oct 31 13:36:43.012911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:43.012846    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d" podStartSLOduration=2.012835616 podStartE2EDuration="2.012835616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:43.011467926 +0000 UTC m=+2000.409816563" watchObservedRunningTime="2026-10-31 13:36:43.012835616 +0000 UTC m=+2000.411184240"
Oct 31 13:36:43.035233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:43.035194    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s" podStartSLOduration=2.035182362 podStartE2EDuration="2.035182362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:43.031192035 +0000 UTC m=+2000.429540655" watchObservedRunningTime="2026-10-31 13:36:43.035182362 +0000 UTC m=+2000.433530980"
Oct 31 13:36:43.076237 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:43.076205    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"]
Oct 31 13:36:43.080681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:43.080654    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-580f24-85c8f7565f-zxkv9"]
Oct 31 13:36:43.613623 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:43.613399    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:43.642230 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:43.642176    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:36:43.791782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:43.791696    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:43.870010 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:43.869252    4428 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 31 13:36:43.870010 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:43.869312    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb" podUID="65d39130-1203-4a53-af59-cd0d7c2afd57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 31 13:36:44.017606 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:44.017566    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d" event={"ID":"b83d1055-13eb-489e-9bc2-17780894bcfb","Type":"ContainerStarted","Data":"516b4378255b44897cd9ea1f5ee64be6705eb738e009eb0fc2f232281ff4ad41"}
Oct 31 13:36:44.760990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:44.760956    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="767378d1-3634-4db5-873f-7cc9fc3fb5a9" path="/var/lib/kubelet/pods/767378d1-3634-4db5-873f-7cc9fc3fb5a9/volumes"
Oct 31 13:36:44.863895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:44.862651    4428 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 31 13:36:44.863895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:44.862717    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb" podUID="65d39130-1203-4a53-af59-cd0d7c2afd57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 31 13:36:45.024534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:45.024452    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:45.024974 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:45.024938    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"]
Oct 31 13:36:45.025199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:45.025176    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy" containerID="cri-o://776373b78bb26d3d9e1d16a0a0aac2ac43c7a172a2f4680260027487b4f7720c" gracePeriod=30
Oct 31 13:36:45.033699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:45.033662    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"]
Oct 31 13:36:45.034064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:45.034037    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy" containerID="cri-o://0d1e14884122ce4152cf1a4fd94da9d5fd81fd968dda9b48d5baab35a82e7cd8" gracePeriod=30
Oct 31 13:36:45.207896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:45.197584    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:45.254437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:45.250634    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"]
Oct 31 13:36:45.254437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:45.250910    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy" containerID="cri-o://5291084f9d7bdedbd61f2509b5e459d37a540487afbd35f270dfcdf434410aee" gracePeriod=30
Oct 31 13:36:45.867807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:45.867771    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:45.868344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:45.868319    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:45.869537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:45.869514    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:46.232891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:46.232559    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:47.429020 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:47.428978    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:48.495996 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:48.494854    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:48.742733 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:48.742699    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:36:49.426176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.426139    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"]
Oct 31 13:36:49.427566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.427539    4428 cpu_manager.go:414] "containerMap: removing container" podUID="767378d1-3634-4db5-873f-7cc9fc3fb5a9" containerName="payload-pre-processing"
Oct 31 13:36:49.427753 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.427735    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767378d1-3634-4db5-873f-7cc9fc3fb5a9" containerName="payload-pre-processing"
Oct 31 13:36:49.428201 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.428181    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="767378d1-3634-4db5-873f-7cc9fc3fb5a9" containerName="payload-pre-processing"
Oct 31 13:36:49.436251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.436168    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 13:36:49.442419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.442387    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"]
Oct 31 13:36:49.496242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.496218    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-5e8f29-serving-cert\""
Oct 31 13:36:49.606888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.606468    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:49.610807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.609258    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eacbe93-226a-4def-96fe-b8409cc76f3b-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws\" (UID: \"2eacbe93-226a-4def-96fe-b8409cc76f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 13:36:49.610807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.609402    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eacbe93-226a-4def-96fe-b8409cc76f3b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws\" (UID: \"2eacbe93-226a-4def-96fe-b8409cc76f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 13:36:49.610807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.609524    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eacbe93-226a-4def-96fe-b8409cc76f3b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws\" (UID: \"2eacbe93-226a-4def-96fe-b8409cc76f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 13:36:49.610807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.609553    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5m6p\" (UniqueName: \"kubernetes.io/projected/2eacbe93-226a-4def-96fe-b8409cc76f3b-kube-api-access-c5m6p\") pod \"maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws\" (UID: \"2eacbe93-226a-4def-96fe-b8409cc76f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 13:36:49.711380 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.711342    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eacbe93-226a-4def-96fe-b8409cc76f3b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws\" (UID: \"2eacbe93-226a-4def-96fe-b8409cc76f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 13:36:49.711643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.711623    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5m6p\" (UniqueName: \"kubernetes.io/projected/2eacbe93-226a-4def-96fe-b8409cc76f3b-kube-api-access-c5m6p\") pod \"maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws\" (UID: \"2eacbe93-226a-4def-96fe-b8409cc76f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 13:36:49.712055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.712034    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eacbe93-226a-4def-96fe-b8409cc76f3b-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws\" (UID: \"2eacbe93-226a-4def-96fe-b8409cc76f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 13:36:49.712271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.712255    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eacbe93-226a-4def-96fe-b8409cc76f3b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws\" (UID: \"2eacbe93-226a-4def-96fe-b8409cc76f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 13:36:49.714840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.714822    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eacbe93-226a-4def-96fe-b8409cc76f3b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws\" (UID: \"2eacbe93-226a-4def-96fe-b8409cc76f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 13:36:49.716937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.716623    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eacbe93-226a-4def-96fe-b8409cc76f3b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws\" (UID: \"2eacbe93-226a-4def-96fe-b8409cc76f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 13:36:49.721448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.721409    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eacbe93-226a-4def-96fe-b8409cc76f3b-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws\" (UID: \"2eacbe93-226a-4def-96fe-b8409cc76f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 13:36:49.733095 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.732679    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5m6p\" (UniqueName: \"kubernetes.io/projected/2eacbe93-226a-4def-96fe-b8409cc76f3b-kube-api-access-c5m6p\") pod \"maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws\" (UID: \"2eacbe93-226a-4def-96fe-b8409cc76f3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 13:36:49.764644 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.764620    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 13:36:49.778045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:49.778023    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:50.120196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.113195    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"]
Oct 31 13:36:50.120196 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:36:50.113348    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eacbe93_226a_4def_96fe_b8409cc76f3b.slice/crio-2b6735b8b8aee68f86bc3d8c6064305d1e5d4db1bb72336d3ad98f2d6d1bc0e2 WatchSource:0}: Error finding container 2b6735b8b8aee68f86bc3d8c6064305d1e5d4db1bb72336d3ad98f2d6d1bc0e2: Status 404 returned error can't find the container with id 2b6735b8b8aee68f86bc3d8c6064305d1e5d4db1bb72336d3ad98f2d6d1bc0e2
Oct 31 13:36:50.488072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.487512    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:36:50.571997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.571703    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-workload-certs\") pod \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") "
Oct 31 13:36:50.571997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.571755    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-envoy\") pod \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") "
Oct 31 13:36:50.571997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.571816    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istiod-ca-cert\") pod \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") "
Oct 31 13:36:50.571997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.571886    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-workload-socket\") pod \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") "
Oct 31 13:36:50.573160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.572615    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istiod-ca-cert" pod "056cecdf-d4f4-44c9-8cc5-416e24b29f86" (UID: "056cecdf-d4f4-44c9-8cc5-416e24b29f86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:50.573160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.572814    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-workload-socket" pod "056cecdf-d4f4-44c9-8cc5-416e24b29f86" (UID: "056cecdf-d4f4-44c9-8cc5-416e24b29f86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:50.573160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.572854    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-workload-certs" pod "056cecdf-d4f4-44c9-8cc5-416e24b29f86" (UID: "056cecdf-d4f4-44c9-8cc5-416e24b29f86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:50.573160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.573052    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056cecdf-d4f4-44c9-8cc5-416e24b29f86-kube-api-access-wp4kb\" (UniqueName: \"kubernetes.io/projected/056cecdf-d4f4-44c9-8cc5-416e24b29f86-kube-api-access-wp4kb\") pod \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") "
Oct 31 13:36:50.573160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.573150    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-podinfo\") pod \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") "
Oct 31 13:36:50.573657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.573638    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.573657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.573657    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.573810 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.573666    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.576076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.576037    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-envoy" pod "056cecdf-d4f4-44c9-8cc5-416e24b29f86" (UID: "056cecdf-d4f4-44c9-8cc5-416e24b29f86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:50.578118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.578092    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056cecdf-d4f4-44c9-8cc5-416e24b29f86-kube-api-access-wp4kb" pod "056cecdf-d4f4-44c9-8cc5-416e24b29f86" (UID: "056cecdf-d4f4-44c9-8cc5-416e24b29f86"). InnerVolumeSpecName "kube-api-access-wp4kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:50.584698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.584667    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-podinfo" pod "056cecdf-d4f4-44c9-8cc5-416e24b29f86" (UID: "056cecdf-d4f4-44c9-8cc5-416e24b29f86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:36:50.656498 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.655966    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.791928    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-data\") pod \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") "
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.792048    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-credential-socket\") pod \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") "
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.792081    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-workload-socket\") pod \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") "
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.792156    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-token\" (UniqueName: \"kubernetes.io/projected/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-token\") pod \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") "
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.792196    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-podinfo\") pod \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") "
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.792247    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-credential-socket\") pod \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") "
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.792291    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-kube-api-access-m8s4b\" (UniqueName: \"kubernetes.io/projected/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-kube-api-access-m8s4b\") pod \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") "
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.792321    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-data" pod "056cecdf-d4f4-44c9-8cc5-416e24b29f86" (UID: "056cecdf-d4f4-44c9-8cc5-416e24b29f86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.792340    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-envoy\") pod \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") "
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.792402    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istiod-ca-cert\") pod \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") "
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.792432    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-token\" (UniqueName: \"kubernetes.io/projected/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-token\") pod \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\" (UID: \"056cecdf-d4f4-44c9-8cc5-416e24b29f86\") "
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.792573    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-workload-certs\") pod \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") "
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.792655    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-data\") pod \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\" (UID: \"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c\") "
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.792748    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-credential-socket" pod "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" (UID: "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.793558    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp4kb\" (UniqueName: \"kubernetes.io/projected/056cecdf-d4f4-44c9-8cc5-416e24b29f86-kube-api-access-wp4kb\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.793582    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.793598    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.793611    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.793625    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.793928    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-data" pod "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" (UID: "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.795299    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-credential-socket" pod "056cecdf-d4f4-44c9-8cc5-416e24b29f86" (UID: "056cecdf-d4f4-44c9-8cc5-416e24b29f86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.795476    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-workload-socket" pod "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" (UID: "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.808536    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istiod-ca-cert" pod "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" (UID: "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:50.825430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.814912    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-workload-certs" pod "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" (UID: "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:50.850851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.850105    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-podinfo" pod "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" (UID: "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:36:50.850851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.850153    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-kube-api-access-m8s4b" pod "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" (UID: "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c"). InnerVolumeSpecName "kube-api-access-m8s4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:50.850851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.850255    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-token" pod "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" (UID: "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:50.858106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.856122    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-token" pod "056cecdf-d4f4-44c9-8cc5-416e24b29f86" (UID: "056cecdf-d4f4-44c9-8cc5-416e24b29f86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:50.858106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.856855    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-envoy" pod "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" (UID: "07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:50.895089 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.895035    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/056cecdf-d4f4-44c9-8cc5-416e24b29f86-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.895089 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.895070    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.895089 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.895087    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.895332 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.895100    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.895332 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.895115    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8s4b\" (UniqueName: \"kubernetes.io/projected/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-kube-api-access-m8s4b\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.895332 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.895131    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.895332 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.895145    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.895332 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.895159    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/056cecdf-d4f4-44c9-8cc5-416e24b29f86-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.895332 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.895170    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.895332 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.895179    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.902789 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.902769    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:36:50.996745 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.996704    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12211035-140d-4934-9c27-36ef18fd6d1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12211035-140d-4934-9c27-36ef18fd6d1b-istiod-ca-cert\") pod \"12211035-140d-4934-9c27-36ef18fd6d1b\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") "
Oct 31 13:36:50.996940 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.996837    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-workload-socket\") pod \"12211035-140d-4934-9c27-36ef18fd6d1b\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") "
Oct 31 13:36:50.997210 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.997009    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12211035-140d-4934-9c27-36ef18fd6d1b-kube-api-access-wg27w\" (UniqueName: \"kubernetes.io/projected/12211035-140d-4934-9c27-36ef18fd6d1b-kube-api-access-wg27w\") pod \"12211035-140d-4934-9c27-36ef18fd6d1b\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") "
Oct 31 13:36:50.997210 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.997051    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-credential-socket\") pod \"12211035-140d-4934-9c27-36ef18fd6d1b\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") "
Oct 31 13:36:50.997210 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.997150    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-istio-envoy\") pod \"12211035-140d-4934-9c27-36ef18fd6d1b\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") "
Oct 31 13:36:50.997210 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.997207    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-workload-certs\") pod \"12211035-140d-4934-9c27-36ef18fd6d1b\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") "
Oct 31 13:36:50.997543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.997241    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-workload-socket" pod "12211035-140d-4934-9c27-36ef18fd6d1b" (UID: "12211035-140d-4934-9c27-36ef18fd6d1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:50.997543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.997278    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12211035-140d-4934-9c27-36ef18fd6d1b-istio-token\" (UniqueName: \"kubernetes.io/projected/12211035-140d-4934-9c27-36ef18fd6d1b-istio-token\") pod \"12211035-140d-4934-9c27-36ef18fd6d1b\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") "
Oct 31 13:36:50.997543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.997311    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12211035-140d-4934-9c27-36ef18fd6d1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12211035-140d-4934-9c27-36ef18fd6d1b-istio-podinfo\") pod \"12211035-140d-4934-9c27-36ef18fd6d1b\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") "
Oct 31 13:36:50.997543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.997372    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-istio-data\") pod \"12211035-140d-4934-9c27-36ef18fd6d1b\" (UID: \"12211035-140d-4934-9c27-36ef18fd6d1b\") "
Oct 31 13:36:50.997543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.997472    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-workload-certs" pod "12211035-140d-4934-9c27-36ef18fd6d1b" (UID: "12211035-140d-4934-9c27-36ef18fd6d1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:50.997908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.997727    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12211035-140d-4934-9c27-36ef18fd6d1b-istiod-ca-cert" pod "12211035-140d-4934-9c27-36ef18fd6d1b" (UID: "12211035-140d-4934-9c27-36ef18fd6d1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:50.997908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.997755    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-credential-socket" pod "12211035-140d-4934-9c27-36ef18fd6d1b" (UID: "12211035-140d-4934-9c27-36ef18fd6d1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:50.998167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.998141    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-istio-data" pod "12211035-140d-4934-9c27-36ef18fd6d1b" (UID: "12211035-140d-4934-9c27-36ef18fd6d1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:50.998543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.998451    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.998543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.998476    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.998543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.998491    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.998543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.998506    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:50.998543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:50.998520    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12211035-140d-4934-9c27-36ef18fd6d1b-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:51.000856 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.000829    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12211035-140d-4934-9c27-36ef18fd6d1b-istio-token" pod "12211035-140d-4934-9c27-36ef18fd6d1b" (UID: "12211035-140d-4934-9c27-36ef18fd6d1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:51.000971 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.000888    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-istio-envoy" pod "12211035-140d-4934-9c27-36ef18fd6d1b" (UID: "12211035-140d-4934-9c27-36ef18fd6d1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:51.000971 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.000931    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12211035-140d-4934-9c27-36ef18fd6d1b-kube-api-access-wg27w" pod "12211035-140d-4934-9c27-36ef18fd6d1b" (UID: "12211035-140d-4934-9c27-36ef18fd6d1b"). InnerVolumeSpecName "kube-api-access-wg27w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:51.001100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.001076    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12211035-140d-4934-9c27-36ef18fd6d1b-istio-podinfo" pod "12211035-140d-4934-9c27-36ef18fd6d1b" (UID: "12211035-140d-4934-9c27-36ef18fd6d1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:36:51.100172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.100140    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12211035-140d-4934-9c27-36ef18fd6d1b-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:51.100172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.100167    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12211035-140d-4934-9c27-36ef18fd6d1b-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:51.100172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.100177    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12211035-140d-4934-9c27-36ef18fd6d1b-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:51.100415 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.100186    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg27w\" (UniqueName: \"kubernetes.io/projected/12211035-140d-4934-9c27-36ef18fd6d1b-kube-api-access-wg27w\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:51.140296 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.140204    4428 generic.go:356] "Generic (PLEG): container finished" podID="12211035-140d-4934-9c27-36ef18fd6d1b" containerID="5291084f9d7bdedbd61f2509b5e459d37a540487afbd35f270dfcdf434410aee" exitCode=0
Oct 31 13:36:51.140296 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.140249    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"
Oct 31 13:36:51.140565 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.140286    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" event={"ID":"12211035-140d-4934-9c27-36ef18fd6d1b","Type":"ContainerDied","Data":"5291084f9d7bdedbd61f2509b5e459d37a540487afbd35f270dfcdf434410aee"}
Oct 31 13:36:51.140565 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.140346    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld" event={"ID":"12211035-140d-4934-9c27-36ef18fd6d1b","Type":"ContainerDied","Data":"92d1dbc2818fc925d0ec7c61887f80c34ca7a95bf64fa7e5d18715f474497e66"}
Oct 31 13:36:51.140565 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.140375    4428 scope.go:122] "RemoveContainer" containerID="5291084f9d7bdedbd61f2509b5e459d37a540487afbd35f270dfcdf434410aee"
Oct 31 13:36:51.142223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.142199    4428 generic.go:356] "Generic (PLEG): container finished" podID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerID="776373b78bb26d3d9e1d16a0a0aac2ac43c7a172a2f4680260027487b4f7720c" exitCode=0
Oct 31 13:36:51.142447 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.142231    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" event={"ID":"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c","Type":"ContainerDied","Data":"776373b78bb26d3d9e1d16a0a0aac2ac43c7a172a2f4680260027487b4f7720c"}
Oct 31 13:36:51.142537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.142499    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj" event={"ID":"07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c","Type":"ContainerDied","Data":"0c9c22f1e261e3e48408dd439aa2a30e0d2f897f13c16d3234bdd430131133f2"}
Oct 31 13:36:51.142537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.142260    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"
Oct 31 13:36:51.144427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.144246    4428 generic.go:356] "Generic (PLEG): container finished" podID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerID="0d1e14884122ce4152cf1a4fd94da9d5fd81fd968dda9b48d5baab35a82e7cd8" exitCode=0
Oct 31 13:36:51.144427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.144303    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"
Oct 31 13:36:51.144427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.144404    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" event={"ID":"056cecdf-d4f4-44c9-8cc5-416e24b29f86","Type":"ContainerDied","Data":"0d1e14884122ce4152cf1a4fd94da9d5fd81fd968dda9b48d5baab35a82e7cd8"}
Oct 31 13:36:51.144651 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.144456    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq" event={"ID":"056cecdf-d4f4-44c9-8cc5-416e24b29f86","Type":"ContainerDied","Data":"5853477e2f558eb5fa5b6883b371a6ae4e9c6b3fd68cb5ea54c0c32ac28dbda6"}
Oct 31 13:36:51.150699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.150125    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws" event={"ID":"2eacbe93-226a-4def-96fe-b8409cc76f3b","Type":"ContainerStarted","Data":"6d07a417a529943aeef6e945444deda779c3c6c95a53e3b20af6386508946cd3"}
Oct 31 13:36:51.150699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.150171    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws" event={"ID":"2eacbe93-226a-4def-96fe-b8409cc76f3b","Type":"ContainerStarted","Data":"2b6735b8b8aee68f86bc3d8c6064305d1e5d4db1bb72336d3ad98f2d6d1bc0e2"}
Oct 31 13:36:51.150699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.150208    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 13:36:51.156930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.156915    4428 scope.go:122] "RemoveContainer" containerID="5291084f9d7bdedbd61f2509b5e459d37a540487afbd35f270dfcdf434410aee"
Oct 31 13:36:51.157280 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:51.157251    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5291084f9d7bdedbd61f2509b5e459d37a540487afbd35f270dfcdf434410aee\": container with ID starting with 5291084f9d7bdedbd61f2509b5e459d37a540487afbd35f270dfcdf434410aee not found: ID does not exist" containerID="5291084f9d7bdedbd61f2509b5e459d37a540487afbd35f270dfcdf434410aee"
Oct 31 13:36:51.157372 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.157288    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5291084f9d7bdedbd61f2509b5e459d37a540487afbd35f270dfcdf434410aee"} err="failed to get container status \"5291084f9d7bdedbd61f2509b5e459d37a540487afbd35f270dfcdf434410aee\": rpc error: code = NotFound desc = could not find container \"5291084f9d7bdedbd61f2509b5e459d37a540487afbd35f270dfcdf434410aee\": container with ID starting with 5291084f9d7bdedbd61f2509b5e459d37a540487afbd35f270dfcdf434410aee not found: ID does not exist"
Oct 31 13:36:51.157372 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.157310    4428 scope.go:122] "RemoveContainer" containerID="776373b78bb26d3d9e1d16a0a0aac2ac43c7a172a2f4680260027487b4f7720c"
Oct 31 13:36:51.171668 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.171651    4428 scope.go:122] "RemoveContainer" containerID="776373b78bb26d3d9e1d16a0a0aac2ac43c7a172a2f4680260027487b4f7720c"
Oct 31 13:36:51.172000 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:51.171981    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776373b78bb26d3d9e1d16a0a0aac2ac43c7a172a2f4680260027487b4f7720c\": container with ID starting with 776373b78bb26d3d9e1d16a0a0aac2ac43c7a172a2f4680260027487b4f7720c not found: ID does not exist" containerID="776373b78bb26d3d9e1d16a0a0aac2ac43c7a172a2f4680260027487b4f7720c"
Oct 31 13:36:51.172067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.172008    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776373b78bb26d3d9e1d16a0a0aac2ac43c7a172a2f4680260027487b4f7720c"} err="failed to get container status \"776373b78bb26d3d9e1d16a0a0aac2ac43c7a172a2f4680260027487b4f7720c\": rpc error: code = NotFound desc = could not find container \"776373b78bb26d3d9e1d16a0a0aac2ac43c7a172a2f4680260027487b4f7720c\": container with ID starting with 776373b78bb26d3d9e1d16a0a0aac2ac43c7a172a2f4680260027487b4f7720c not found: ID does not exist"
Oct 31 13:36:51.172067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.172031    4428 scope.go:122] "RemoveContainer" containerID="0d1e14884122ce4152cf1a4fd94da9d5fd81fd968dda9b48d5baab35a82e7cd8"
Oct 31 13:36:51.181794 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.181773    4428 scope.go:122] "RemoveContainer" containerID="0d1e14884122ce4152cf1a4fd94da9d5fd81fd968dda9b48d5baab35a82e7cd8"
Oct 31 13:36:51.182070 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:51.182046    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1e14884122ce4152cf1a4fd94da9d5fd81fd968dda9b48d5baab35a82e7cd8\": container with ID starting with 0d1e14884122ce4152cf1a4fd94da9d5fd81fd968dda9b48d5baab35a82e7cd8 not found: ID does not exist" containerID="0d1e14884122ce4152cf1a4fd94da9d5fd81fd968dda9b48d5baab35a82e7cd8"
Oct 31 13:36:51.182149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.182080    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1e14884122ce4152cf1a4fd94da9d5fd81fd968dda9b48d5baab35a82e7cd8"} err="failed to get container status \"0d1e14884122ce4152cf1a4fd94da9d5fd81fd968dda9b48d5baab35a82e7cd8\": rpc error: code = NotFound desc = could not find container \"0d1e14884122ce4152cf1a4fd94da9d5fd81fd968dda9b48d5baab35a82e7cd8\": container with ID starting with 0d1e14884122ce4152cf1a4fd94da9d5fd81fd968dda9b48d5baab35a82e7cd8 not found: ID does not exist"
Oct 31 13:36:51.191728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.191691    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws" podStartSLOduration=1.757018178 podStartE2EDuration="2.19168031s" totalImagesPullingTime="434.662132ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:51.188990618 +0000 UTC m=+2008.587339242" watchObservedRunningTime="2026-10-31 13:36:51.19168031 +0000 UTC m=+2008.590028934"
Oct 31 13:36:51.214162 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.214138    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"]
Oct 31 13:36:51.224541 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.224516    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-5m5bq"]
Oct 31 13:36:51.240251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.240225    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"]
Oct 31 13:36:51.244995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.244974    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-jhchj"]
Oct 31 13:36:51.260629 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.260606    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"]
Oct 31 13:36:51.267454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.267433    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-hdhld"]
Oct 31 13:36:51.299862 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:51.299836    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:52.052210 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.052124    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"]
Oct 31 13:36:52.053343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.053316    4428 cpu_manager.go:414] "containerMap: removing container" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy"
Oct 31 13:36:52.053343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.053344    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy"
Oct 31 13:36:52.053552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.053382    4428 cpu_manager.go:414] "containerMap: removing container" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy"
Oct 31 13:36:52.053552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.053391    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy"
Oct 31 13:36:52.053552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.053408    4428 cpu_manager.go:414] "containerMap: removing container" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy"
Oct 31 13:36:52.053552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.053415    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy"
Oct 31 13:36:52.053759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.053628    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" containerName="istio-proxy"
Oct 31 13:36:52.053759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.053645    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" containerName="istio-proxy"
Oct 31 13:36:52.053759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.053667    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" containerName="istio-proxy"
Oct 31 13:36:52.059428 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.059405    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.086070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.086043    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"]
Oct 31 13:36:52.117652 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.117620    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.117790 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.117754    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h4s7\" (UniqueName: \"kubernetes.io/projected/c6602f73-021d-4155-9972-037349415d91-kube-api-access-5h4s7\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.117963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.117942    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.118018 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.118003    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6602f73-021d-4155-9972-037349415d91-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.118085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.118035    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6602f73-021d-4155-9972-037349415d91-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.118166 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.118147    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.118208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.118181    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.118289 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.118275    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.118342 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.118325    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6602f73-021d-4155-9972-037349415d91-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.240946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.232304    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.240946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.232366    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6602f73-021d-4155-9972-037349415d91-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.240946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.232394    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6602f73-021d-4155-9972-037349415d91-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.240946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.232503    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.240946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.232530    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.240946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.232631    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.240946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.232692    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6602f73-021d-4155-9972-037349415d91-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.240946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.232837    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.240946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.232995    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h4s7\" (UniqueName: \"kubernetes.io/projected/c6602f73-021d-4155-9972-037349415d91-kube-api-access-5h4s7\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.240946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.237203    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.240946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.238380    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6602f73-021d-4155-9972-037349415d91-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.240946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.238645    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.240946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.239037    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.240946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.239351    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.252742 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.242244    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.252742 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.251254    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:52.252742 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.251632    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h4s7\" (UniqueName: \"kubernetes.io/projected/c6602f73-021d-4155-9972-037349415d91-kube-api-access-5h4s7\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.269518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.268315    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6602f73-021d-4155-9972-037349415d91-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.293559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.283284    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6602f73-021d-4155-9972-037349415d91-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.387274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.387199    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:52.684628 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:36:52.684596    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6602f73_021d_4155_9972_037349415d91.slice/crio-4bde63c93ab319180dbe2cc83338d48f6bf90ff19218f8ddba7b9a440b560f8f WatchSource:0}: Error finding container 4bde63c93ab319180dbe2cc83338d48f6bf90ff19218f8ddba7b9a440b560f8f: Status 404 returned error can't find the container with id 4bde63c93ab319180dbe2cc83338d48f6bf90ff19218f8ddba7b9a440b560f8f
Oct 31 13:36:52.685820 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.685791    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"]
Oct 31 13:36:52.687541 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.687510    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:52.687620 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.687579    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:52.687620 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.687608    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:36:52.757903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.757858    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056cecdf-d4f4-44c9-8cc5-416e24b29f86" path="/var/lib/kubelet/pods/056cecdf-d4f4-44c9-8cc5-416e24b29f86/volumes"
Oct 31 13:36:52.758422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.758406    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c" path="/var/lib/kubelet/pods/07a8d6f8-00bd-42b5-8eb8-3b5053cf2e8c/volumes"
Oct 31 13:36:52.758851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:52.758838    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12211035-140d-4934-9c27-36ef18fd6d1b" path="/var/lib/kubelet/pods/12211035-140d-4934-9c27-36ef18fd6d1b/volumes"
Oct 31 13:36:53.172497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:53.172458    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj" event={"ID":"c6602f73-021d-4155-9972-037349415d91","Type":"ContainerStarted","Data":"c83a52b4931f680d3f0e4a86de17daf4c7d7a8f083532ca353b9bbcc45e5f277"}
Oct 31 13:36:53.172977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:53.172523    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj" event={"ID":"c6602f73-021d-4155-9972-037349415d91","Type":"ContainerStarted","Data":"4bde63c93ab319180dbe2cc83338d48f6bf90ff19218f8ddba7b9a440b560f8f"}
Oct 31 13:36:53.199960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:53.199900    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj" podStartSLOduration=1.199861739 podStartE2EDuration="1.199861739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:53.197442457 +0000 UTC m=+2010.595791122" watchObservedRunningTime="2026-10-31 13:36:53.199861739 +0000 UTC m=+2010.598210365"
Oct 31 13:36:53.307028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:53.306991    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:53.395257 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:53.395229    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:36:53.402049 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:53.402011    4428 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 31 13:36:53.402318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:53.402289    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj" podUID="c6602f73-021d-4155-9972-037349415d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 31 13:36:53.740234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:53.740205    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:53.919324 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:53.919276    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:54.021580 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:54.021495    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"]
Oct 31 13:36:54.021777 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:54.021744    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb" podUID="65d39130-1203-4a53-af59-cd0d7c2afd57" containerName="istio-proxy" containerID="cri-o://4a085768ec72537a1fc02bbf9aef27a06b48d3bd50af84d8625d8fb4a0517c80" gracePeriod=30
Oct 31 13:36:54.029621 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:54.029588    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"]
Oct 31 13:36:54.035272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:54.035245    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"
Oct 31 13:36:54.041388 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:54.041359    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"]
Oct 31 13:36:54.041602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:54.041576    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz" podUID="05c3c622-9d4a-4391-bb25-e07daa031d3b" containerName="istio-proxy" containerID="cri-o://c6436d1bd8b649a7275752cb416dbd0d827de5c5426a882c80d1df4da4ae5703" gracePeriod=30
Oct 31 13:36:54.118896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:54.117401    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:54.327054 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:54.325436    4428 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj" secret="" err="secret \"e2e-mt-79f4fb0f-openshift-default-dockercfg-shgv6\" not found"
Oct 31 13:36:54.421894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:54.392289    4428 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 31 13:36:54.421894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:54.392354    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj" podUID="c6602f73-021d-4155-9972-037349415d91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 31 13:36:54.967445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:54.967407    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:55.044986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:55.044959    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"
Oct 31 13:36:55.334282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:55.334199    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj" podUID="c6602f73-021d-4155-9972-037349415d91" containerName="istio-proxy" containerID="cri-o://c83a52b4931f680d3f0e4a86de17daf4c7d7a8f083532ca353b9bbcc45e5f277" gracePeriod=30
Oct 31 13:36:56.181447 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:56.181386    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:57.088904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:57.079464    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:57.222894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:57.222552    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 13:36:58.659890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:58.654760    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:59.352176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.352153    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:36:59.382021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.381998    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:59.425500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.425463    4428 generic.go:356] "Generic (PLEG): container finished" podID="05c3c622-9d4a-4391-bb25-e07daa031d3b" containerID="c6436d1bd8b649a7275752cb416dbd0d827de5c5426a882c80d1df4da4ae5703" exitCode=0
Oct 31 13:36:59.425804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.425778    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz" event={"ID":"05c3c622-9d4a-4391-bb25-e07daa031d3b","Type":"ContainerDied","Data":"c6436d1bd8b649a7275752cb416dbd0d827de5c5426a882c80d1df4da4ae5703"}
Oct 31 13:36:59.425996 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.425977    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz" event={"ID":"05c3c622-9d4a-4391-bb25-e07daa031d3b","Type":"ContainerDied","Data":"62a79ed32f6517f81caf53dfc74b4e2ccbb0246323aea0da79e3badb44799089"}
Oct 31 13:36:59.426151 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.426136    4428 scope.go:122] "RemoveContainer" containerID="c6436d1bd8b649a7275752cb416dbd0d827de5c5426a882c80d1df4da4ae5703"
Oct 31 13:36:59.426415 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.426396    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"
Oct 31 13:36:59.430368 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.430344    4428 generic.go:356] "Generic (PLEG): container finished" podID="65d39130-1203-4a53-af59-cd0d7c2afd57" containerID="4a085768ec72537a1fc02bbf9aef27a06b48d3bd50af84d8625d8fb4a0517c80" exitCode=0
Oct 31 13:36:59.430642 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.430618    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb" event={"ID":"65d39130-1203-4a53-af59-cd0d7c2afd57","Type":"ContainerDied","Data":"4a085768ec72537a1fc02bbf9aef27a06b48d3bd50af84d8625d8fb4a0517c80"}
Oct 31 13:36:59.430815 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.430787    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb" event={"ID":"65d39130-1203-4a53-af59-cd0d7c2afd57","Type":"ContainerDied","Data":"6774d8c64da365beea86299f935fb4b6f3d34a79e93ccfa2e727fcfc0f66bacf"}
Oct 31 13:36:59.431045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.431028    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"
Oct 31 13:36:59.448153 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.448132    4428 scope.go:122] "RemoveContainer" containerID="d878d22f219f9da551c49546ba7589ec1ed1c0d1884cf56056ec8fa180e6f05a"
Oct 31 13:36:59.461654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.461630    4428 scope.go:122] "RemoveContainer" containerID="c6436d1bd8b649a7275752cb416dbd0d827de5c5426a882c80d1df4da4ae5703"
Oct 31 13:36:59.462163 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:59.462137    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6436d1bd8b649a7275752cb416dbd0d827de5c5426a882c80d1df4da4ae5703\": container with ID starting with c6436d1bd8b649a7275752cb416dbd0d827de5c5426a882c80d1df4da4ae5703 not found: ID does not exist" containerID="c6436d1bd8b649a7275752cb416dbd0d827de5c5426a882c80d1df4da4ae5703"
Oct 31 13:36:59.462264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.462176    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6436d1bd8b649a7275752cb416dbd0d827de5c5426a882c80d1df4da4ae5703"} err="failed to get container status \"c6436d1bd8b649a7275752cb416dbd0d827de5c5426a882c80d1df4da4ae5703\": rpc error: code = NotFound desc = could not find container \"c6436d1bd8b649a7275752cb416dbd0d827de5c5426a882c80d1df4da4ae5703\": container with ID starting with c6436d1bd8b649a7275752cb416dbd0d827de5c5426a882c80d1df4da4ae5703 not found: ID does not exist"
Oct 31 13:36:59.462264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.462201    4428 scope.go:122] "RemoveContainer" containerID="d878d22f219f9da551c49546ba7589ec1ed1c0d1884cf56056ec8fa180e6f05a"
Oct 31 13:36:59.462553 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:59.462526    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d878d22f219f9da551c49546ba7589ec1ed1c0d1884cf56056ec8fa180e6f05a\": container with ID starting with d878d22f219f9da551c49546ba7589ec1ed1c0d1884cf56056ec8fa180e6f05a not found: ID does not exist" containerID="d878d22f219f9da551c49546ba7589ec1ed1c0d1884cf56056ec8fa180e6f05a"
Oct 31 13:36:59.462615 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.462562    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d878d22f219f9da551c49546ba7589ec1ed1c0d1884cf56056ec8fa180e6f05a"} err="failed to get container status \"d878d22f219f9da551c49546ba7589ec1ed1c0d1884cf56056ec8fa180e6f05a\": rpc error: code = NotFound desc = could not find container \"d878d22f219f9da551c49546ba7589ec1ed1c0d1884cf56056ec8fa180e6f05a\": container with ID starting with d878d22f219f9da551c49546ba7589ec1ed1c0d1884cf56056ec8fa180e6f05a not found: ID does not exist"
Oct 31 13:36:59.462615 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.462584    4428 scope.go:122] "RemoveContainer" containerID="4a085768ec72537a1fc02bbf9aef27a06b48d3bd50af84d8625d8fb4a0517c80"
Oct 31 13:36:59.481964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.481944    4428 scope.go:122] "RemoveContainer" containerID="4a085768ec72537a1fc02bbf9aef27a06b48d3bd50af84d8625d8fb4a0517c80"
Oct 31 13:36:59.482253 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:36:59.482229    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a085768ec72537a1fc02bbf9aef27a06b48d3bd50af84d8625d8fb4a0517c80\": container with ID starting with 4a085768ec72537a1fc02bbf9aef27a06b48d3bd50af84d8625d8fb4a0517c80 not found: ID does not exist" containerID="4a085768ec72537a1fc02bbf9aef27a06b48d3bd50af84d8625d8fb4a0517c80"
Oct 31 13:36:59.482323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.482263    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a085768ec72537a1fc02bbf9aef27a06b48d3bd50af84d8625d8fb4a0517c80"} err="failed to get container status \"4a085768ec72537a1fc02bbf9aef27a06b48d3bd50af84d8625d8fb4a0517c80\": rpc error: code = NotFound desc = could not find container \"4a085768ec72537a1fc02bbf9aef27a06b48d3bd50af84d8625d8fb4a0517c80\": container with ID starting with 4a085768ec72537a1fc02bbf9aef27a06b48d3bd50af84d8625d8fb4a0517c80 not found: ID does not exist"
Oct 31 13:36:59.533779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.533743    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-token\" (UniqueName: \"kubernetes.io/projected/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-token\") pod \"05c3c622-9d4a-4391-bb25-e07daa031d3b\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") "
Oct 31 13:36:59.533982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.533802    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d39130-1203-4a53-af59-cd0d7c2afd57-kube-api-access-drspc\" (UniqueName: \"kubernetes.io/projected/65d39130-1203-4a53-af59-cd0d7c2afd57-kube-api-access-drspc\") pod \"65d39130-1203-4a53-af59-cd0d7c2afd57\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") "
Oct 31 13:36:59.533982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.533846    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-credential-socket\") pod \"65d39130-1203-4a53-af59-cd0d7c2afd57\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") "
Oct 31 13:36:59.533982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.533930    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-workload-socket\") pod \"05c3c622-9d4a-4391-bb25-e07daa031d3b\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") "
Oct 31 13:36:59.533982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.533962    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c3c622-9d4a-4391-bb25-e07daa031d3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c3c622-9d4a-4391-bb25-e07daa031d3b-istiod-ca-cert\") pod \"05c3c622-9d4a-4391-bb25-e07daa031d3b\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") "
Oct 31 13:36:59.534260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.534000    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-envoy\") pod \"05c3c622-9d4a-4391-bb25-e07daa031d3b\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") "
Oct 31 13:36:59.534260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.534054    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65d39130-1203-4a53-af59-cd0d7c2afd57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d39130-1203-4a53-af59-cd0d7c2afd57-istiod-ca-cert\") pod \"65d39130-1203-4a53-af59-cd0d7c2afd57\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") "
Oct 31 13:36:59.534260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.534087    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-credential-socket\") pod \"05c3c622-9d4a-4391-bb25-e07daa031d3b\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") "
Oct 31 13:36:59.534260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.534153    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-podinfo\") pod \"65d39130-1203-4a53-af59-cd0d7c2afd57\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") "
Oct 31 13:36:59.534260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.534197    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-data\") pod \"05c3c622-9d4a-4391-bb25-e07daa031d3b\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") "
Oct 31 13:36:59.534260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.534229    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-workload-certs\") pod \"65d39130-1203-4a53-af59-cd0d7c2afd57\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") "
Oct 31 13:36:59.534604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.534374    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-token\" (UniqueName: \"kubernetes.io/projected/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-token\") pod \"65d39130-1203-4a53-af59-cd0d7c2afd57\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") "
Oct 31 13:36:59.534604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.534441    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-envoy\") pod \"65d39130-1203-4a53-af59-cd0d7c2afd57\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") "
Oct 31 13:36:59.534604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.534485    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c3c622-9d4a-4391-bb25-e07daa031d3b-kube-api-access-jbcsq\" (UniqueName: \"kubernetes.io/projected/05c3c622-9d4a-4391-bb25-e07daa031d3b-kube-api-access-jbcsq\") pod \"05c3c622-9d4a-4391-bb25-e07daa031d3b\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") "
Oct 31 13:36:59.534604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.534525    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-podinfo\") pod \"05c3c622-9d4a-4391-bb25-e07daa031d3b\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") "
Oct 31 13:36:59.534604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.534594    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-workload-certs\") pod \"05c3c622-9d4a-4391-bb25-e07daa031d3b\" (UID: \"05c3c622-9d4a-4391-bb25-e07daa031d3b\") "
Oct 31 13:36:59.534903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.534702    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-workload-socket\") pod \"65d39130-1203-4a53-af59-cd0d7c2afd57\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") "
Oct 31 13:36:59.534903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.534809    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-data\") pod \"65d39130-1203-4a53-af59-cd0d7c2afd57\" (UID: \"65d39130-1203-4a53-af59-cd0d7c2afd57\") "
Oct 31 13:36:59.535832 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.535803    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-data" pod "65d39130-1203-4a53-af59-cd0d7c2afd57" (UID: "65d39130-1203-4a53-af59-cd0d7c2afd57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:59.537571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.537541    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-data" pod "05c3c622-9d4a-4391-bb25-e07daa031d3b" (UID: "05c3c622-9d4a-4391-bb25-e07daa031d3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:59.537848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.537825    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c3c622-9d4a-4391-bb25-e07daa031d3b-istiod-ca-cert" pod "05c3c622-9d4a-4391-bb25-e07daa031d3b" (UID: "05c3c622-9d4a-4391-bb25-e07daa031d3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:59.538243 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.538217    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-workload-certs" pod "65d39130-1203-4a53-af59-cd0d7c2afd57" (UID: "65d39130-1203-4a53-af59-cd0d7c2afd57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:59.538693 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.538534    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-workload-certs" pod "05c3c622-9d4a-4391-bb25-e07daa031d3b" (UID: "05c3c622-9d4a-4391-bb25-e07daa031d3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:59.539055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.538847    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-workload-socket" pod "65d39130-1203-4a53-af59-cd0d7c2afd57" (UID: "65d39130-1203-4a53-af59-cd0d7c2afd57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:59.541807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.541644    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-workload-socket" pod "05c3c622-9d4a-4391-bb25-e07daa031d3b" (UID: "05c3c622-9d4a-4391-bb25-e07daa031d3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:59.541807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.541776    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-credential-socket" pod "05c3c622-9d4a-4391-bb25-e07daa031d3b" (UID: "05c3c622-9d4a-4391-bb25-e07daa031d3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:59.542150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.542125    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-credential-socket" pod "65d39130-1203-4a53-af59-cd0d7c2afd57" (UID: "65d39130-1203-4a53-af59-cd0d7c2afd57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:59.542150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.542144    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d39130-1203-4a53-af59-cd0d7c2afd57-istiod-ca-cert" pod "65d39130-1203-4a53-af59-cd0d7c2afd57" (UID: "65d39130-1203-4a53-af59-cd0d7c2afd57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:36:59.543499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.543394    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-token" pod "05c3c622-9d4a-4391-bb25-e07daa031d3b" (UID: "05c3c622-9d4a-4391-bb25-e07daa031d3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:59.543499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.543475    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-envoy" pod "05c3c622-9d4a-4391-bb25-e07daa031d3b" (UID: "05c3c622-9d4a-4391-bb25-e07daa031d3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:59.549181 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.547267    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c3c622-9d4a-4391-bb25-e07daa031d3b-kube-api-access-jbcsq" pod "05c3c622-9d4a-4391-bb25-e07daa031d3b" (UID: "05c3c622-9d4a-4391-bb25-e07daa031d3b"). InnerVolumeSpecName "kube-api-access-jbcsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:59.549181 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.547402    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-token" pod "65d39130-1203-4a53-af59-cd0d7c2afd57" (UID: "65d39130-1203-4a53-af59-cd0d7c2afd57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:59.549181 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.547502    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-podinfo" pod "65d39130-1203-4a53-af59-cd0d7c2afd57" (UID: "65d39130-1203-4a53-af59-cd0d7c2afd57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:36:59.549181 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.547551    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-podinfo" pod "05c3c622-9d4a-4391-bb25-e07daa031d3b" (UID: "05c3c622-9d4a-4391-bb25-e07daa031d3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:36:59.549181 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.549094    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-envoy" pod "65d39130-1203-4a53-af59-cd0d7c2afd57" (UID: "65d39130-1203-4a53-af59-cd0d7c2afd57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:36:59.549720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.549698    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d39130-1203-4a53-af59-cd0d7c2afd57-kube-api-access-drspc" pod "65d39130-1203-4a53-af59-cd0d7c2afd57" (UID: "65d39130-1203-4a53-af59-cd0d7c2afd57"). InnerVolumeSpecName "kube-api-access-drspc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:36:59.637990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.637905    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.637990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.637941    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.637990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.637957    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbcsq\" (UniqueName: \"kubernetes.io/projected/05c3c622-9d4a-4391-bb25-e07daa031d3b-kube-api-access-jbcsq\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.637990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.637973    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.637990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.637987    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.638255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.638000    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.638255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.638015    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.638255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.638028    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.638255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.638043    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drspc\" (UniqueName: \"kubernetes.io/projected/65d39130-1203-4a53-af59-cd0d7c2afd57-kube-api-access-drspc\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.638255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.638057    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.638255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.638070    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.638255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.638083    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c3c622-9d4a-4391-bb25-e07daa031d3b-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.638255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.638096    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.638255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.638108    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d39130-1203-4a53-af59-cd0d7c2afd57-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.638255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.638122    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.638255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.638135    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d39130-1203-4a53-af59-cd0d7c2afd57-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.638255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.638148    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c3c622-9d4a-4391-bb25-e07daa031d3b-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.638255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.638160    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d39130-1203-4a53-af59-cd0d7c2afd57-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:36:59.649739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.649704    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:36:59.851523 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.851449    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"]
Oct 31 13:36:59.906531 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.882809    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-77cgz"]
Oct 31 13:36:59.921628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:36:59.920537    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"]
Oct 31 13:37:00.001906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:00.001827    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-wzxnb"]
Oct 31 13:37:00.356516 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:00.350421    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"]
Oct 31 13:37:00.356516 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:00.350714    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" podUID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" containerName="istio-proxy" containerID="cri-o://144cde08a76993f7629edf6d1db20383968cf953baba73f8cb83aee2a1bc7146" gracePeriod=30
Oct 31 13:37:00.415735 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:00.415665    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:00.568420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:00.568386    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"]
Oct 31 13:37:00.568654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:00.568628    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x" podUID="04337deb-7a21-4e58-8a41-098d8be2e918" containerName="istio-proxy" containerID="cri-o://80928b5c5ee0345e1495fb1e3d18877c09c8fa337571687312e97d43a0fc1123" gracePeriod=30
Oct 31 13:37:00.577612 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:00.577586    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"]
Oct 31 13:37:00.577807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:00.577784    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" containerID="cri-o://f5919dc9db36dd8b4abdd55cfabb93aa3353b39b8ee3b7812b4bf511fbd213a4" gracePeriod=30
Oct 31 13:37:00.582680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:00.582656    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"]
Oct 31 13:37:00.583513 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:00.583480    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst" podUID="f5b5a82a-0de7-4c45-94a5-b8c046ea9387" containerName="istio-proxy" containerID="cri-o://f161292e68d5d1ccd5e62af7417e5a70b5171e7af061eb453aee43bda43a9f62" gracePeriod=30
Oct 31 13:37:00.688768 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:00.688682    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:00.763069 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:00.763034    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c3c622-9d4a-4391-bb25-e07daa031d3b" path="/var/lib/kubelet/pods/05c3c622-9d4a-4391-bb25-e07daa031d3b/volumes"
Oct 31 13:37:00.763659 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:00.763640    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d39130-1203-4a53-af59-cd0d7c2afd57" path="/var/lib/kubelet/pods/65d39130-1203-4a53-af59-cd0d7c2afd57/volumes"
Oct 31 13:37:00.951836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:00.951681    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:01.013337 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.013304    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"]
Oct 31 13:37:01.022396 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.022361    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cce3e7-vfdq7"]
Oct 31 13:37:01.291090 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.291005    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"]
Oct 31 13:37:01.292061 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.292040    4428 cpu_manager.go:414] "containerMap: removing container" podUID="05c3c622-9d4a-4391-bb25-e07daa031d3b" containerName="istio-proxy"
Oct 31 13:37:01.292061 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.292059    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c3c622-9d4a-4391-bb25-e07daa031d3b" containerName="istio-proxy"
Oct 31 13:37:01.292246 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.292096    4428 cpu_manager.go:414] "containerMap: removing container" podUID="65d39130-1203-4a53-af59-cd0d7c2afd57" containerName="istio-proxy"
Oct 31 13:37:01.292246 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.292104    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d39130-1203-4a53-af59-cd0d7c2afd57" containerName="istio-proxy"
Oct 31 13:37:01.292246 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.292125    4428 cpu_manager.go:414] "containerMap: removing container" podUID="05c3c622-9d4a-4391-bb25-e07daa031d3b" containerName="istio-proxy"
Oct 31 13:37:01.292246 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.292131    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c3c622-9d4a-4391-bb25-e07daa031d3b" containerName="istio-proxy"
Oct 31 13:37:01.292373 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.292252    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d39130-1203-4a53-af59-cd0d7c2afd57" containerName="istio-proxy"
Oct 31 13:37:01.292373 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.292265    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c3c622-9d4a-4391-bb25-e07daa031d3b" containerName="istio-proxy"
Oct 31 13:37:01.292373 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.292274    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c3c622-9d4a-4391-bb25-e07daa031d3b" containerName="istio-proxy"
Oct 31 13:37:01.297663 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.297635    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"
Oct 31 13:37:01.308537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.308510    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"]
Oct 31 13:37:01.354226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.354174    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5j2s\" (UniqueName: \"kubernetes.io/projected/60c24e33-d236-4bb6-adcb-3c2160c30b8d-kube-api-access-s5j2s\") pod \"maas-api-revoke-keys-e2e-api-a-6aa924-6lx56\" (UID: \"60c24e33-d236-4bb6-adcb-3c2160c30b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"
Oct 31 13:37:01.356006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.355970    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c24e33-d236-4bb6-adcb-3c2160c30b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6aa924-6lx56\" (UID: \"60c24e33-d236-4bb6-adcb-3c2160c30b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"
Oct 31 13:37:01.356411 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.356322    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c24e33-d236-4bb6-adcb-3c2160c30b8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6aa924-6lx56\" (UID: \"60c24e33-d236-4bb6-adcb-3c2160c30b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"
Oct 31 13:37:01.359477 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.359452    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:01.360176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.360157    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:37:01.360585 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.360565    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:37:01.468949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.465260    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5j2s\" (UniqueName: \"kubernetes.io/projected/60c24e33-d236-4bb6-adcb-3c2160c30b8d-kube-api-access-s5j2s\") pod \"maas-api-revoke-keys-e2e-api-a-6aa924-6lx56\" (UID: \"60c24e33-d236-4bb6-adcb-3c2160c30b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"
Oct 31 13:37:01.479138 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.479104    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c24e33-d236-4bb6-adcb-3c2160c30b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6aa924-6lx56\" (UID: \"60c24e33-d236-4bb6-adcb-3c2160c30b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"
Oct 31 13:37:01.479313 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.479170    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c24e33-d236-4bb6-adcb-3c2160c30b8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6aa924-6lx56\" (UID: \"60c24e33-d236-4bb6-adcb-3c2160c30b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"
Oct 31 13:37:01.480438 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.480402    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c24e33-d236-4bb6-adcb-3c2160c30b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6aa924-6lx56\" (UID: \"60c24e33-d236-4bb6-adcb-3c2160c30b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"
Oct 31 13:37:01.487693 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.487217    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c24e33-d236-4bb6-adcb-3c2160c30b8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6aa924-6lx56\" (UID: \"60c24e33-d236-4bb6-adcb-3c2160c30b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"
Oct 31 13:37:01.492565 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.492489    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5j2s\" (UniqueName: \"kubernetes.io/projected/60c24e33-d236-4bb6-adcb-3c2160c30b8d-kube-api-access-s5j2s\") pod \"maas-api-revoke-keys-e2e-api-a-6aa924-6lx56\" (UID: \"60c24e33-d236-4bb6-adcb-3c2160c30b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"
Oct 31 13:37:01.536475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.536439    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:01.620818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.620736    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"
Oct 31 13:37:01.909957 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:37:01.909927    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c24e33_d236_4bb6_adcb_3c2160c30b8d.slice/crio-8d7030b7d063ec9d98f2086db8960cae30e55a4cb007d3abcf9325b8ec1400a8 WatchSource:0}: Error finding container 8d7030b7d063ec9d98f2086db8960cae30e55a4cb007d3abcf9325b8ec1400a8: Status 404 returned error can't find the container with id 8d7030b7d063ec9d98f2086db8960cae30e55a4cb007d3abcf9325b8ec1400a8
Oct 31 13:37:01.910528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:01.910492    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"]
Oct 31 13:37:02.579848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.579766    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56" event={"ID":"60c24e33-d236-4bb6-adcb-3c2160c30b8d","Type":"ContainerStarted","Data":"1cdb4eb5d2d9f9a9b579c1078ce8537a4bed16697b50b1a117b26c07cbb39727"}
Oct 31 13:37:02.579848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.579831    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56" event={"ID":"60c24e33-d236-4bb6-adcb-3c2160c30b8d","Type":"ContainerStarted","Data":"8d7030b7d063ec9d98f2086db8960cae30e55a4cb007d3abcf9325b8ec1400a8"}
Oct 31 13:37:02.604513 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.604468    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56" podStartSLOduration=1.604453994 podStartE2EDuration="1.604453994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:02.600935207 +0000 UTC m=+2019.999283847" watchObservedRunningTime="2026-10-31 13:37:02.604453994 +0000 UTC m=+2020.002802617"
Oct 31 13:37:02.652605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.652576    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"]
Oct 31 13:37:02.658612 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.658588    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:37:02.669836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.669810    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-69bdf5-serving-cert\""
Oct 31 13:37:02.678234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.678206    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"]
Oct 31 13:37:02.742106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.742074    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5hzf\" (UniqueName: \"kubernetes.io/projected/c6533b62-cafe-4af5-a32d-2bb882ebe41b-kube-api-access-z5hzf\") pod \"maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd\" (UID: \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:37:02.742266 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.742135    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6533b62-cafe-4af5-a32d-2bb882ebe41b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd\" (UID: \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:37:02.742266 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.742231    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6533b62-cafe-4af5-a32d-2bb882ebe41b-metrics-certs\") pod \"maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd\" (UID: \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:37:02.742390 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.742283    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6533b62-cafe-4af5-a32d-2bb882ebe41b-maas-api-tls\") pod \"maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd\" (UID: \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:37:02.758075 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.758048    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c934f1a7-3712-4955-ad79-5505b3760938" path="/var/lib/kubelet/pods/c934f1a7-3712-4955-ad79-5505b3760938/volumes"
Oct 31 13:37:02.770601 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.770578    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:02.855950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.852593    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5hzf\" (UniqueName: \"kubernetes.io/projected/c6533b62-cafe-4af5-a32d-2bb882ebe41b-kube-api-access-z5hzf\") pod \"maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd\" (UID: \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:37:02.874282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.872058    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6533b62-cafe-4af5-a32d-2bb882ebe41b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd\" (UID: \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:37:02.874282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.872167    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6533b62-cafe-4af5-a32d-2bb882ebe41b-metrics-certs\") pod \"maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd\" (UID: \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:37:02.874282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.872261    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6533b62-cafe-4af5-a32d-2bb882ebe41b-maas-api-tls\") pod \"maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd\" (UID: \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:37:02.874282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.873829    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6533b62-cafe-4af5-a32d-2bb882ebe41b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd\" (UID: \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:37:02.895259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.895220    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6533b62-cafe-4af5-a32d-2bb882ebe41b-maas-api-tls\") pod \"maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd\" (UID: \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:37:02.901892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.899369    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6533b62-cafe-4af5-a32d-2bb882ebe41b-metrics-certs\") pod \"maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd\" (UID: \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:37:02.933895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.929231    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5hzf\" (UniqueName: \"kubernetes.io/projected/c6533b62-cafe-4af5-a32d-2bb882ebe41b-kube-api-access-z5hzf\") pod \"maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd\" (UID: \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:37:02.946894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.945789    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:02.983099 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:02.983055    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:37:03.277776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:03.277749    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"]
Oct 31 13:37:03.278615 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:37:03.278590    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6533b62_cafe_4af5_a32d_2bb882ebe41b.slice/crio-571af416397612c1176e48039f027ee606d6372ae54755cd7948fd3516f315d4 WatchSource:0}: Error finding container 571af416397612c1176e48039f027ee606d6372ae54755cd7948fd3516f315d4: Status 404 returned error can't find the container with id 571af416397612c1176e48039f027ee606d6372ae54755cd7948fd3516f315d4
Oct 31 13:37:03.589744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:03.589660    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd" event={"ID":"c6533b62-cafe-4af5-a32d-2bb882ebe41b","Type":"ContainerStarted","Data":"571af416397612c1176e48039f027ee606d6372ae54755cd7948fd3516f315d4"}
Oct 31 13:37:03.591440 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:03.591411    4428 generic.go:356] "Generic (PLEG): container finished" podID="60c24e33-d236-4bb6-adcb-3c2160c30b8d" containerID="1cdb4eb5d2d9f9a9b579c1078ce8537a4bed16697b50b1a117b26c07cbb39727" exitCode=0
Oct 31 13:37:03.591567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:03.591449    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56" event={"ID":"60c24e33-d236-4bb6-adcb-3c2160c30b8d","Type":"ContainerDied","Data":"1cdb4eb5d2d9f9a9b579c1078ce8537a4bed16697b50b1a117b26c07cbb39727"}
Oct 31 13:37:04.297989 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:04.287546    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:04.727892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:04.725859    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd" event={"ID":"c6533b62-cafe-4af5-a32d-2bb882ebe41b","Type":"ContainerStarted","Data":"600f760be36d4e5925fa614d63f3a39abf47196fd01ba82f8787d2c2f21ea4f0"}
Oct 31 13:37:04.727892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:04.726009    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:37:04.819901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:04.819773    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd" podStartSLOduration=2.128474198 podStartE2EDuration="2.819749654s" totalImagesPullingTime="691.275456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:04.75125024 +0000 UTC m=+2022.149598865" watchObservedRunningTime="2026-10-31 13:37:04.819749654 +0000 UTC m=+2022.218098278"
Oct 31 13:37:05.328096 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.323566    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-vcml7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:37:05.328096 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.323625    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7" podUID="0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:37:05.671065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.648509    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"
Oct 31 13:37:05.740073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.739554    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60c24e33-d236-4bb6-adcb-3c2160c30b8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c24e33-d236-4bb6-adcb-3c2160c30b8d-maas-api-xks-ca\") pod \"60c24e33-d236-4bb6-adcb-3c2160c30b8d\" (UID: \"60c24e33-d236-4bb6-adcb-3c2160c30b8d\") "
Oct 31 13:37:05.740073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.739826    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c24e33-d236-4bb6-adcb-3c2160c30b8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c24e33-d236-4bb6-adcb-3c2160c30b8d-maas-api-openshift-ca\") pod \"60c24e33-d236-4bb6-adcb-3c2160c30b8d\" (UID: \"60c24e33-d236-4bb6-adcb-3c2160c30b8d\") "
Oct 31 13:37:05.741958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.740733    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c24e33-d236-4bb6-adcb-3c2160c30b8d-kube-api-access-s5j2s\" (UniqueName: \"kubernetes.io/projected/60c24e33-d236-4bb6-adcb-3c2160c30b8d-kube-api-access-s5j2s\") pod \"60c24e33-d236-4bb6-adcb-3c2160c30b8d\" (UID: \"60c24e33-d236-4bb6-adcb-3c2160c30b8d\") "
Oct 31 13:37:05.752103 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.751274    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c24e33-d236-4bb6-adcb-3c2160c30b8d-kube-api-access-s5j2s" pod "60c24e33-d236-4bb6-adcb-3c2160c30b8d" (UID: "60c24e33-d236-4bb6-adcb-3c2160c30b8d"). InnerVolumeSpecName "kube-api-access-s5j2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:05.752103 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.751356    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c24e33-d236-4bb6-adcb-3c2160c30b8d-maas-api-xks-ca" pod "60c24e33-d236-4bb6-adcb-3c2160c30b8d" (UID: "60c24e33-d236-4bb6-adcb-3c2160c30b8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:37:05.779890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.774752    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c24e33-d236-4bb6-adcb-3c2160c30b8d-maas-api-openshift-ca" pod "60c24e33-d236-4bb6-adcb-3c2160c30b8d" (UID: "60c24e33-d236-4bb6-adcb-3c2160c30b8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:05.824898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.818573    4428 generic.go:356] "Generic (PLEG): container finished" podID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" containerID="144cde08a76993f7629edf6d1db20383968cf953baba73f8cb83aee2a1bc7146" exitCode=0
Oct 31 13:37:05.824898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.818757    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" event={"ID":"170b8b28-9f7f-494f-b6e6-86d5af6dcfec","Type":"ContainerDied","Data":"144cde08a76993f7629edf6d1db20383968cf953baba73f8cb83aee2a1bc7146"}
Oct 31 13:37:05.847888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.845168    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c24e33-d236-4bb6-adcb-3c2160c30b8d-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:05.847888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.845200    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c24e33-d236-4bb6-adcb-3c2160c30b8d-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:05.847888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.845217    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5j2s\" (UniqueName: \"kubernetes.io/projected/60c24e33-d236-4bb6-adcb-3c2160c30b8d-kube-api-access-s5j2s\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:05.860924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.855435    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56" event={"ID":"60c24e33-d236-4bb6-adcb-3c2160c30b8d","Type":"ContainerDied","Data":"8d7030b7d063ec9d98f2086db8960cae30e55a4cb007d3abcf9325b8ec1400a8"}
Oct 31 13:37:05.860924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.855499    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d7030b7d063ec9d98f2086db8960cae30e55a4cb007d3abcf9325b8ec1400a8"
Oct 31 13:37:05.860924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.855579    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"
Oct 31 13:37:05.866359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.865160    4428 generic.go:356] "Generic (PLEG): container finished" podID="c6602f73-021d-4155-9972-037349415d91" containerID="c83a52b4931f680d3f0e4a86de17daf4c7d7a8f083532ca353b9bbcc45e5f277" exitCode=0
Oct 31 13:37:05.866359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.866014    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj" event={"ID":"c6602f73-021d-4155-9972-037349415d91","Type":"ContainerDied","Data":"c83a52b4931f680d3f0e4a86de17daf4c7d7a8f083532ca353b9bbcc45e5f277"}
Oct 31 13:37:05.962553 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:05.962105    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:37:06.066714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.066684    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-istio-data\") pod \"c6602f73-021d-4155-9972-037349415d91\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") "
Oct 31 13:37:06.066957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.066944    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6602f73-021d-4155-9972-037349415d91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6602f73-021d-4155-9972-037349415d91-istio-podinfo\") pod \"c6602f73-021d-4155-9972-037349415d91\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") "
Oct 31 13:37:06.067049 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.067041    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6602f73-021d-4155-9972-037349415d91-istio-token\" (UniqueName: \"kubernetes.io/projected/c6602f73-021d-4155-9972-037349415d91-istio-token\") pod \"c6602f73-021d-4155-9972-037349415d91\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") "
Oct 31 13:37:06.067165 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.067156    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-credential-socket\") pod \"c6602f73-021d-4155-9972-037349415d91\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") "
Oct 31 13:37:06.067255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.067247    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-workload-certs\") pod \"c6602f73-021d-4155-9972-037349415d91\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") "
Oct 31 13:37:06.067632 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.067605    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-workload-certs" pod "c6602f73-021d-4155-9972-037349415d91" (UID: "c6602f73-021d-4155-9972-037349415d91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.068173 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.068126    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-credential-socket" pod "c6602f73-021d-4155-9972-037349415d91" (UID: "c6602f73-021d-4155-9972-037349415d91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.068277 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.068256    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-istio-envoy\") pod \"c6602f73-021d-4155-9972-037349415d91\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") "
Oct 31 13:37:06.069830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.068537    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6602f73-021d-4155-9972-037349415d91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6602f73-021d-4155-9972-037349415d91-istiod-ca-cert\") pod \"c6602f73-021d-4155-9972-037349415d91\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") "
Oct 31 13:37:06.069830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.068591    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-workload-socket\") pod \"c6602f73-021d-4155-9972-037349415d91\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") "
Oct 31 13:37:06.069830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.068625    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6602f73-021d-4155-9972-037349415d91-kube-api-access-5h4s7\" (UniqueName: \"kubernetes.io/projected/c6602f73-021d-4155-9972-037349415d91-kube-api-access-5h4s7\") pod \"c6602f73-021d-4155-9972-037349415d91\" (UID: \"c6602f73-021d-4155-9972-037349415d91\") "
Oct 31 13:37:06.069830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.069081    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6602f73-021d-4155-9972-037349415d91-istiod-ca-cert" pod "c6602f73-021d-4155-9972-037349415d91" (UID: "c6602f73-021d-4155-9972-037349415d91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:06.069830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.069493    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.069830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.069515    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.069830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.069529    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6602f73-021d-4155-9972-037349415d91-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.069830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.069786    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-workload-socket" pod "c6602f73-021d-4155-9972-037349415d91" (UID: "c6602f73-021d-4155-9972-037349415d91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.076341 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.076275    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6602f73-021d-4155-9972-037349415d91-istio-token" pod "c6602f73-021d-4155-9972-037349415d91" (UID: "c6602f73-021d-4155-9972-037349415d91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:06.076615 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.076571    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-istio-envoy" pod "c6602f73-021d-4155-9972-037349415d91" (UID: "c6602f73-021d-4155-9972-037349415d91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.078378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.078348    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6602f73-021d-4155-9972-037349415d91-kube-api-access-5h4s7" pod "c6602f73-021d-4155-9972-037349415d91" (UID: "c6602f73-021d-4155-9972-037349415d91"). InnerVolumeSpecName "kube-api-access-5h4s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:06.078499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.078478    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6602f73-021d-4155-9972-037349415d91-istio-podinfo" pod "c6602f73-021d-4155-9972-037349415d91" (UID: "c6602f73-021d-4155-9972-037349415d91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:37:06.084646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.084610    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-istio-data" pod "c6602f73-021d-4155-9972-037349415d91" (UID: "c6602f73-021d-4155-9972-037349415d91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.107015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.106550    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:37:06.175310 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.175272    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-workload-socket\") pod \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") "
Oct 31 13:37:06.175659 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.175634    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istiod-ca-cert\") pod \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") "
Oct 31 13:37:06.175752 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.175710    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-envoy\") pod \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") "
Oct 31 13:37:06.175957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.175552    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-workload-socket" pod "170b8b28-9f7f-494f-b6e6-86d5af6dcfec" (UID: "170b8b28-9f7f-494f-b6e6-86d5af6dcfec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.176070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.175959    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-data\") pod \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") "
Oct 31 13:37:06.176070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.176000    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-workload-certs\") pod \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") "
Oct 31 13:37:06.176181 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.176075    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-podinfo\") pod \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") "
Oct 31 13:37:06.176251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.176193    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-token\" (UniqueName: \"kubernetes.io/projected/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-token\") pod \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") "
Oct 31 13:37:06.176251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.176233    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-credential-socket\") pod \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") "
Oct 31 13:37:06.176481 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.176306    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-data" pod "170b8b28-9f7f-494f-b6e6-86d5af6dcfec" (UID: "170b8b28-9f7f-494f-b6e6-86d5af6dcfec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.176557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.176425    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istiod-ca-cert" pod "170b8b28-9f7f-494f-b6e6-86d5af6dcfec" (UID: "170b8b28-9f7f-494f-b6e6-86d5af6dcfec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:06.176557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.176321    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-kube-api-access-dxd5v\" (UniqueName: \"kubernetes.io/projected/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-kube-api-access-dxd5v\") pod \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\" (UID: \"170b8b28-9f7f-494f-b6e6-86d5af6dcfec\") "
Oct 31 13:37:06.177015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.176988    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-workload-certs" pod "170b8b28-9f7f-494f-b6e6-86d5af6dcfec" (UID: "170b8b28-9f7f-494f-b6e6-86d5af6dcfec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.177379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.177354    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.177482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.177383    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.177482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.177399    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.177482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.177413    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h4s7\" (UniqueName: \"kubernetes.io/projected/c6602f73-021d-4155-9972-037349415d91-kube-api-access-5h4s7\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.177482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.177427    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.177482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.177440    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6602f73-021d-4155-9972-037349415d91-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.177482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.177454    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6602f73-021d-4155-9972-037349415d91-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.177482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.177469    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6602f73-021d-4155-9972-037349415d91-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.177482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.177481    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.177948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.177496    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.177948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.177710    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-credential-socket" pod "170b8b28-9f7f-494f-b6e6-86d5af6dcfec" (UID: "170b8b28-9f7f-494f-b6e6-86d5af6dcfec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.180372 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.180342    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-kube-api-access-dxd5v" pod "170b8b28-9f7f-494f-b6e6-86d5af6dcfec" (UID: "170b8b28-9f7f-494f-b6e6-86d5af6dcfec"). InnerVolumeSpecName "kube-api-access-dxd5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:06.180496 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.180459    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-token" pod "170b8b28-9f7f-494f-b6e6-86d5af6dcfec" (UID: "170b8b28-9f7f-494f-b6e6-86d5af6dcfec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:06.182748 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.182710    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-envoy" pod "170b8b28-9f7f-494f-b6e6-86d5af6dcfec" (UID: "170b8b28-9f7f-494f-b6e6-86d5af6dcfec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.186938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.186129    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-podinfo" pod "170b8b28-9f7f-494f-b6e6-86d5af6dcfec" (UID: "170b8b28-9f7f-494f-b6e6-86d5af6dcfec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:37:06.195541 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.195513    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 13:37:06.195655 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.195566    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 13:37:06.208716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.207511    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:37:06.278899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.278828    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-envoy\") pod \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") "
Oct 31 13:37:06.279084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.278917    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-credential-socket\") pod \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") "
Oct 31 13:37:06.279084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.278959    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-workload-socket\") pod \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") "
Oct 31 13:37:06.279084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.279078    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-workload-certs\") pod \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") "
Oct 31 13:37:06.279303 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.279161    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istiod-ca-cert\") pod \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") "
Oct 31 13:37:06.279303 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.279227    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-token\" (UniqueName: \"kubernetes.io/projected/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-token\") pod \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") "
Oct 31 13:37:06.279303 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.279288    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-podinfo\") pod \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") "
Oct 31 13:37:06.279452 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.279355    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-kube-api-access-8m258\" (UniqueName: \"kubernetes.io/projected/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-kube-api-access-8m258\") pod \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") "
Oct 31 13:37:06.279452 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.279432    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-data\") pod \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\" (UID: \"f5b5a82a-0de7-4c45-94a5-b8c046ea9387\") "
Oct 31 13:37:06.280262 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.280235    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.280409 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.280384    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-credential-socket" pod "f5b5a82a-0de7-4c45-94a5-b8c046ea9387" (UID: "f5b5a82a-0de7-4c45-94a5-b8c046ea9387"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.280512 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.280495    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.280648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.280618    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.280773 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.280656    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxd5v\" (UniqueName: \"kubernetes.io/projected/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-kube-api-access-dxd5v\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.280773 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.280676    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/170b8b28-9f7f-494f-b6e6-86d5af6dcfec-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.280946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.280925    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-data" pod "f5b5a82a-0de7-4c45-94a5-b8c046ea9387" (UID: "f5b5a82a-0de7-4c45-94a5-b8c046ea9387"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.281143 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.281106    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-workload-socket" pod "f5b5a82a-0de7-4c45-94a5-b8c046ea9387" (UID: "f5b5a82a-0de7-4c45-94a5-b8c046ea9387"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.281394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.281367    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istiod-ca-cert" pod "f5b5a82a-0de7-4c45-94a5-b8c046ea9387" (UID: "f5b5a82a-0de7-4c45-94a5-b8c046ea9387"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:06.281956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.281933    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-workload-certs" pod "f5b5a82a-0de7-4c45-94a5-b8c046ea9387" (UID: "f5b5a82a-0de7-4c45-94a5-b8c046ea9387"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.284967 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.284942    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-envoy" pod "f5b5a82a-0de7-4c45-94a5-b8c046ea9387" (UID: "f5b5a82a-0de7-4c45-94a5-b8c046ea9387"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.285827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.285197    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-podinfo" pod "f5b5a82a-0de7-4c45-94a5-b8c046ea9387" (UID: "f5b5a82a-0de7-4c45-94a5-b8c046ea9387"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:37:06.285827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.285338    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-token" pod "f5b5a82a-0de7-4c45-94a5-b8c046ea9387" (UID: "f5b5a82a-0de7-4c45-94a5-b8c046ea9387"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:06.285827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.285461    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-kube-api-access-8m258" pod "f5b5a82a-0de7-4c45-94a5-b8c046ea9387" (UID: "f5b5a82a-0de7-4c45-94a5-b8c046ea9387"). InnerVolumeSpecName "kube-api-access-8m258". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:06.351922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.351891    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:37:06.381879 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.381839    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.381879 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.381883    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.382065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.381894    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.382065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.381904    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.382065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.381912    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.382065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.381921    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.382065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.381929    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.382065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.381937    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.382065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.381947    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m258\" (UniqueName: \"kubernetes.io/projected/f5b5a82a-0de7-4c45-94a5-b8c046ea9387-kube-api-access-8m258\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.388242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.388212    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:37:06.482940 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.482897    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-podinfo\") pod \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") "
Oct 31 13:37:06.483133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.482985    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-token\" (UniqueName: \"kubernetes.io/projected/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-token\") pod \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") "
Oct 31 13:37:06.483133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483076    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-workload-socket\") pod \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") "
Oct 31 13:37:06.483133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483111    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04337deb-7a21-4e58-8a41-098d8be2e918-istio-token\" (UniqueName: \"kubernetes.io/projected/04337deb-7a21-4e58-8a41-098d8be2e918-istio-token\") pod \"04337deb-7a21-4e58-8a41-098d8be2e918\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") "
Oct 31 13:37:06.483525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483166    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-istio-data\") pod \"04337deb-7a21-4e58-8a41-098d8be2e918\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") "
Oct 31 13:37:06.483525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483211    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istiod-ca-cert\") pod \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") "
Oct 31 13:37:06.483525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483265    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a514fea-055a-47ce-baa7-cd30cb8a88b4-kube-api-access-rf6dk\" (UniqueName: \"kubernetes.io/projected/4a514fea-055a-47ce-baa7-cd30cb8a88b4-kube-api-access-rf6dk\") pod \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") "
Oct 31 13:37:06.483525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483309    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04337deb-7a21-4e58-8a41-098d8be2e918-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04337deb-7a21-4e58-8a41-098d8be2e918-istiod-ca-cert\") pod \"04337deb-7a21-4e58-8a41-098d8be2e918\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") "
Oct 31 13:37:06.483525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483345    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04337deb-7a21-4e58-8a41-098d8be2e918-kube-api-access-554xx\" (UniqueName: \"kubernetes.io/projected/04337deb-7a21-4e58-8a41-098d8be2e918-kube-api-access-554xx\") pod \"04337deb-7a21-4e58-8a41-098d8be2e918\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") "
Oct 31 13:37:06.483525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483395    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-credential-socket\") pod \"04337deb-7a21-4e58-8a41-098d8be2e918\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") "
Oct 31 13:37:06.483525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483449    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-workload-socket\") pod \"04337deb-7a21-4e58-8a41-098d8be2e918\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") "
Oct 31 13:37:06.483934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483561    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-workload-certs\") pod \"04337deb-7a21-4e58-8a41-098d8be2e918\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") "
Oct 31 13:37:06.483934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483548    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-workload-socket" pod "4a514fea-055a-47ce-baa7-cd30cb8a88b4" (UID: "4a514fea-055a-47ce-baa7-cd30cb8a88b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.483934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483602    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-workload-certs\") pod \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") "
Oct 31 13:37:06.483934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483661    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-istio-envoy\") pod \"04337deb-7a21-4e58-8a41-098d8be2e918\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") "
Oct 31 13:37:06.483934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483713    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-data\") pod \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") "
Oct 31 13:37:06.483934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483770    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04337deb-7a21-4e58-8a41-098d8be2e918-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04337deb-7a21-4e58-8a41-098d8be2e918-istio-podinfo\") pod \"04337deb-7a21-4e58-8a41-098d8be2e918\" (UID: \"04337deb-7a21-4e58-8a41-098d8be2e918\") "
Oct 31 13:37:06.483934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483814    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-envoy\") pod \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") "
Oct 31 13:37:06.483934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483848    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-credential-socket\") pod \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\" (UID: \"4a514fea-055a-47ce-baa7-cd30cb8a88b4\") "
Oct 31 13:37:06.485571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.483658    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-istio-data" pod "04337deb-7a21-4e58-8a41-098d8be2e918" (UID: "04337deb-7a21-4e58-8a41-098d8be2e918"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.485571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.484366    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-data" pod "4a514fea-055a-47ce-baa7-cd30cb8a88b4" (UID: "4a514fea-055a-47ce-baa7-cd30cb8a88b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.485571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.484634    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-credential-socket" pod "4a514fea-055a-47ce-baa7-cd30cb8a88b4" (UID: "4a514fea-055a-47ce-baa7-cd30cb8a88b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.485571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.484715    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-credential-socket" pod "04337deb-7a21-4e58-8a41-098d8be2e918" (UID: "04337deb-7a21-4e58-8a41-098d8be2e918"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.485571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.484757    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04337deb-7a21-4e58-8a41-098d8be2e918-istiod-ca-cert" pod "04337deb-7a21-4e58-8a41-098d8be2e918" (UID: "04337deb-7a21-4e58-8a41-098d8be2e918"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:06.485571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.484794    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.485571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.484818    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.485571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.484834    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.485571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.484848    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.485571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.485041    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-workload-socket" pod "04337deb-7a21-4e58-8a41-098d8be2e918" (UID: "04337deb-7a21-4e58-8a41-098d8be2e918"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.485571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.485126    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-workload-certs" pod "04337deb-7a21-4e58-8a41-098d8be2e918" (UID: "04337deb-7a21-4e58-8a41-098d8be2e918"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.485571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.485362    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-workload-certs" pod "4a514fea-055a-47ce-baa7-cd30cb8a88b4" (UID: "4a514fea-055a-47ce-baa7-cd30cb8a88b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.485571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.485414    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istiod-ca-cert" pod "4a514fea-055a-47ce-baa7-cd30cb8a88b4" (UID: "4a514fea-055a-47ce-baa7-cd30cb8a88b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:06.486360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.485648    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-podinfo" pod "4a514fea-055a-47ce-baa7-cd30cb8a88b4" (UID: "4a514fea-055a-47ce-baa7-cd30cb8a88b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:37:06.487454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.487427    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04337deb-7a21-4e58-8a41-098d8be2e918-istio-token" pod "04337deb-7a21-4e58-8a41-098d8be2e918" (UID: "04337deb-7a21-4e58-8a41-098d8be2e918"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:06.487599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.487562    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04337deb-7a21-4e58-8a41-098d8be2e918-kube-api-access-554xx" pod "04337deb-7a21-4e58-8a41-098d8be2e918" (UID: "04337deb-7a21-4e58-8a41-098d8be2e918"). InnerVolumeSpecName "kube-api-access-554xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:06.487599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.487578    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-envoy" pod "4a514fea-055a-47ce-baa7-cd30cb8a88b4" (UID: "4a514fea-055a-47ce-baa7-cd30cb8a88b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.487857 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.487842    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-token" pod "4a514fea-055a-47ce-baa7-cd30cb8a88b4" (UID: "4a514fea-055a-47ce-baa7-cd30cb8a88b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:06.488044 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.488027    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-istio-envoy" pod "04337deb-7a21-4e58-8a41-098d8be2e918" (UID: "04337deb-7a21-4e58-8a41-098d8be2e918"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:37:06.488638 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.488606    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a514fea-055a-47ce-baa7-cd30cb8a88b4-kube-api-access-rf6dk" pod "4a514fea-055a-47ce-baa7-cd30cb8a88b4" (UID: "4a514fea-055a-47ce-baa7-cd30cb8a88b4"). InnerVolumeSpecName "kube-api-access-rf6dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:06.488993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.488972    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04337deb-7a21-4e58-8a41-098d8be2e918-istio-podinfo" pod "04337deb-7a21-4e58-8a41-098d8be2e918" (UID: "04337deb-7a21-4e58-8a41-098d8be2e918"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:37:06.586034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.585997    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.586034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.586024    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.586034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.586037    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.586323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.586050    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04337deb-7a21-4e58-8a41-098d8be2e918-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.586323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.586064    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.586323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.586079    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.586323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.586089    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.586323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.586100    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04337deb-7a21-4e58-8a41-098d8be2e918-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.586323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.586112    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a514fea-055a-47ce-baa7-cd30cb8a88b4-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.586323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.586124    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf6dk\" (UniqueName: \"kubernetes.io/projected/4a514fea-055a-47ce-baa7-cd30cb8a88b4-kube-api-access-rf6dk\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.586323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.586136    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04337deb-7a21-4e58-8a41-098d8be2e918-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.586323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.586148    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-554xx\" (UniqueName: \"kubernetes.io/projected/04337deb-7a21-4e58-8a41-098d8be2e918-kube-api-access-554xx\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.586323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.586161    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.586323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.586175    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04337deb-7a21-4e58-8a41-098d8be2e918-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:06.875062 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.875024    4428 generic.go:356] "Generic (PLEG): container finished" podID="04337deb-7a21-4e58-8a41-098d8be2e918" containerID="80928b5c5ee0345e1495fb1e3d18877c09c8fa337571687312e97d43a0fc1123" exitCode=0
Oct 31 13:37:06.875498 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.875069    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"
Oct 31 13:37:06.875498 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.875098    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x" event={"ID":"04337deb-7a21-4e58-8a41-098d8be2e918","Type":"ContainerDied","Data":"80928b5c5ee0345e1495fb1e3d18877c09c8fa337571687312e97d43a0fc1123"}
Oct 31 13:37:06.875498 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.875153    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x" event={"ID":"04337deb-7a21-4e58-8a41-098d8be2e918","Type":"ContainerDied","Data":"354a325808963882998d74a64cc90cbc72fda1ad58fc8abe7daae9aedea26c8a"}
Oct 31 13:37:06.875498 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.875182    4428 scope.go:122] "RemoveContainer" containerID="80928b5c5ee0345e1495fb1e3d18877c09c8fa337571687312e97d43a0fc1123"
Oct 31 13:37:06.876818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.876773    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"
Oct 31 13:37:06.876957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.876932    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj" event={"ID":"c6602f73-021d-4155-9972-037349415d91","Type":"ContainerDied","Data":"4bde63c93ab319180dbe2cc83338d48f6bf90ff19218f8ddba7b9a440b560f8f"}
Oct 31 13:37:06.878719 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.878695    4428 generic.go:356] "Generic (PLEG): container finished" podID="f5b5a82a-0de7-4c45-94a5-b8c046ea9387" containerID="f161292e68d5d1ccd5e62af7417e5a70b5171e7af061eb453aee43bda43a9f62" exitCode=0
Oct 31 13:37:06.878818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.878745    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"
Oct 31 13:37:06.878818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.878778    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst" event={"ID":"f5b5a82a-0de7-4c45-94a5-b8c046ea9387","Type":"ContainerDied","Data":"f161292e68d5d1ccd5e62af7417e5a70b5171e7af061eb453aee43bda43a9f62"}
Oct 31 13:37:06.879007 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.878831    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst" event={"ID":"f5b5a82a-0de7-4c45-94a5-b8c046ea9387","Type":"ContainerDied","Data":"93e24e46a69f157d347908b6b0b6d643a5ec5be814ee45cbd214a4474eb87129"}
Oct 31 13:37:06.880608 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.880589    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"
Oct 31 13:37:06.880709 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.880615    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh" event={"ID":"170b8b28-9f7f-494f-b6e6-86d5af6dcfec","Type":"ContainerDied","Data":"b19ba9d5b62e9322241e20971093865c63ece300d25bce3f12375a3c70ea2edf"}
Oct 31 13:37:06.882431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.882411    4428 generic.go:356] "Generic (PLEG): container finished" podID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerID="f5919dc9db36dd8b4abdd55cfabb93aa3353b39b8ee3b7812b4bf511fbd213a4" exitCode=0
Oct 31 13:37:06.882608 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.882584    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" event={"ID":"4a514fea-055a-47ce-baa7-cd30cb8a88b4","Type":"ContainerDied","Data":"f5919dc9db36dd8b4abdd55cfabb93aa3353b39b8ee3b7812b4bf511fbd213a4"}
Oct 31 13:37:06.882702 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.882630    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2" event={"ID":"4a514fea-055a-47ce-baa7-cd30cb8a88b4","Type":"ContainerDied","Data":"dc3a86766f291b4e4f264c0fc27d8ab3d92ee70b82ff9ea25add14f867ee7565"}
Oct 31 13:37:06.882846 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.882823    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"
Oct 31 13:37:06.896002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.895976    4428 scope.go:122] "RemoveContainer" containerID="80928b5c5ee0345e1495fb1e3d18877c09c8fa337571687312e97d43a0fc1123"
Oct 31 13:37:06.896337 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:37:06.896304    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80928b5c5ee0345e1495fb1e3d18877c09c8fa337571687312e97d43a0fc1123\": container with ID starting with 80928b5c5ee0345e1495fb1e3d18877c09c8fa337571687312e97d43a0fc1123 not found: ID does not exist" containerID="80928b5c5ee0345e1495fb1e3d18877c09c8fa337571687312e97d43a0fc1123"
Oct 31 13:37:06.896433 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.896341    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80928b5c5ee0345e1495fb1e3d18877c09c8fa337571687312e97d43a0fc1123"} err="failed to get container status \"80928b5c5ee0345e1495fb1e3d18877c09c8fa337571687312e97d43a0fc1123\": rpc error: code = NotFound desc = could not find container \"80928b5c5ee0345e1495fb1e3d18877c09c8fa337571687312e97d43a0fc1123\": container with ID starting with 80928b5c5ee0345e1495fb1e3d18877c09c8fa337571687312e97d43a0fc1123 not found: ID does not exist"
Oct 31 13:37:06.896433 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.896365    4428 scope.go:122] "RemoveContainer" containerID="c83a52b4931f680d3f0e4a86de17daf4c7d7a8f083532ca353b9bbcc45e5f277"
Oct 31 13:37:06.907179 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.907155    4428 scope.go:122] "RemoveContainer" containerID="f161292e68d5d1ccd5e62af7417e5a70b5171e7af061eb453aee43bda43a9f62"
Oct 31 13:37:06.918509 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.918435    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"]
Oct 31 13:37:06.955903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.929524    4428 scope.go:122] "RemoveContainer" containerID="f161292e68d5d1ccd5e62af7417e5a70b5171e7af061eb453aee43bda43a9f62"
Oct 31 13:37:06.955903 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:37:06.930763    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f161292e68d5d1ccd5e62af7417e5a70b5171e7af061eb453aee43bda43a9f62\": container with ID starting with f161292e68d5d1ccd5e62af7417e5a70b5171e7af061eb453aee43bda43a9f62 not found: ID does not exist" containerID="f161292e68d5d1ccd5e62af7417e5a70b5171e7af061eb453aee43bda43a9f62"
Oct 31 13:37:06.955903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.930796    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f161292e68d5d1ccd5e62af7417e5a70b5171e7af061eb453aee43bda43a9f62"} err="failed to get container status \"f161292e68d5d1ccd5e62af7417e5a70b5171e7af061eb453aee43bda43a9f62\": rpc error: code = NotFound desc = could not find container \"f161292e68d5d1ccd5e62af7417e5a70b5171e7af061eb453aee43bda43a9f62\": container with ID starting with f161292e68d5d1ccd5e62af7417e5a70b5171e7af061eb453aee43bda43a9f62 not found: ID does not exist"
Oct 31 13:37:06.955903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.930821    4428 scope.go:122] "RemoveContainer" containerID="144cde08a76993f7629edf6d1db20383968cf953baba73f8cb83aee2a1bc7146"
Oct 31 13:37:06.955903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.937028    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-p6wk2"]
Oct 31 13:37:06.991010 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:06.989441    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:07.034963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:07.026808    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"]
Oct 31 13:37:07.084291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:07.084264    4428 scope.go:122] "RemoveContainer" containerID="f5919dc9db36dd8b4abdd55cfabb93aa3353b39b8ee3b7812b4bf511fbd213a4"
Oct 31 13:37:07.090692 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:07.090660    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-49cst"]
Oct 31 13:37:07.101488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:07.101457    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"]
Oct 31 13:37:07.104685 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:07.104656    4428 scope.go:122] "RemoveContainer" containerID="f5919dc9db36dd8b4abdd55cfabb93aa3353b39b8ee3b7812b4bf511fbd213a4"
Oct 31 13:37:07.105772 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:37:07.105743    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5919dc9db36dd8b4abdd55cfabb93aa3353b39b8ee3b7812b4bf511fbd213a4\": container with ID starting with f5919dc9db36dd8b4abdd55cfabb93aa3353b39b8ee3b7812b4bf511fbd213a4 not found: ID does not exist" containerID="f5919dc9db36dd8b4abdd55cfabb93aa3353b39b8ee3b7812b4bf511fbd213a4"
Oct 31 13:37:07.105890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:07.105783    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5919dc9db36dd8b4abdd55cfabb93aa3353b39b8ee3b7812b4bf511fbd213a4"} err="failed to get container status \"f5919dc9db36dd8b4abdd55cfabb93aa3353b39b8ee3b7812b4bf511fbd213a4\": rpc error: code = NotFound desc = could not find container \"f5919dc9db36dd8b4abdd55cfabb93aa3353b39b8ee3b7812b4bf511fbd213a4\": container with ID starting with f5919dc9db36dd8b4abdd55cfabb93aa3353b39b8ee3b7812b4bf511fbd213a4 not found: ID does not exist"
Oct 31 13:37:07.108628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:07.108603    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-rk4xj"]
Oct 31 13:37:07.113423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:07.113392    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"]
Oct 31 13:37:07.117102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:07.117081    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6xmkh"]
Oct 31 13:37:07.144711 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:07.144680    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"]
Oct 31 13:37:07.153639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:07.153613    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-5dm7x"]
Oct 31 13:37:08.050470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:08.050439    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"]
Oct 31 13:37:08.060594 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:08.060569    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d7ee51-j2nfl"]
Oct 31 13:37:08.761222 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:08.761188    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:08.775949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:08.775804    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04337deb-7a21-4e58-8a41-098d8be2e918" path="/var/lib/kubelet/pods/04337deb-7a21-4e58-8a41-098d8be2e918/volumes"
Oct 31 13:37:08.777251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:08.777201    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd" path="/var/lib/kubelet/pods/0aeeeb1c-3cec-4d8a-b7c4-edfaf70bfefd/volumes"
Oct 31 13:37:08.778385 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:08.778332    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" path="/var/lib/kubelet/pods/170b8b28-9f7f-494f-b6e6-86d5af6dcfec/volumes"
Oct 31 13:37:08.784371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:08.779939    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" path="/var/lib/kubelet/pods/4a514fea-055a-47ce-baa7-cd30cb8a88b4/volumes"
Oct 31 13:37:08.784371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:08.781133    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6602f73-021d-4155-9972-037349415d91" path="/var/lib/kubelet/pods/c6602f73-021d-4155-9972-037349415d91/volumes"
Oct 31 13:37:08.784371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:08.781924    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b5a82a-0de7-4c45-94a5-b8c046ea9387" path="/var/lib/kubelet/pods/f5b5a82a-0de7-4c45-94a5-b8c046ea9387/volumes"
Oct 31 13:37:09.073387 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:09.073298    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"]
Oct 31 13:37:09.085794 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:09.085768    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-728deca0-shldl"]
Oct 31 13:37:09.915893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:09.909411    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:10.757698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:10.757659    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4" path="/var/lib/kubelet/pods/1d1a7f6d-f8a7-4a29-81e2-f4bb51498fe4/volumes"
Oct 31 13:37:10.911615 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:10.911553    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:11.038114 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.038044    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"]
Oct 31 13:37:11.038754 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.038740    4428 cpu_manager.go:414] "containerMap: removing container" podUID="f5b5a82a-0de7-4c45-94a5-b8c046ea9387" containerName="istio-proxy"
Oct 31 13:37:11.038800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.038756    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b5a82a-0de7-4c45-94a5-b8c046ea9387" containerName="istio-proxy"
Oct 31 13:37:11.038800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.038768    4428 cpu_manager.go:414] "containerMap: removing container" podUID="04337deb-7a21-4e58-8a41-098d8be2e918" containerName="istio-proxy"
Oct 31 13:37:11.038800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.038774    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04337deb-7a21-4e58-8a41-098d8be2e918" containerName="istio-proxy"
Oct 31 13:37:11.038800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.038794    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy"
Oct 31 13:37:11.038800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.038799    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy"
Oct 31 13:37:11.038977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.038813    4428 cpu_manager.go:414] "containerMap: removing container" podUID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" containerName="istio-proxy"
Oct 31 13:37:11.038977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.038818    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" containerName="istio-proxy"
Oct 31 13:37:11.038977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.038833    4428 cpu_manager.go:414] "containerMap: removing container" podUID="c6602f73-021d-4155-9972-037349415d91" containerName="istio-proxy"
Oct 31 13:37:11.038977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.038838    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6602f73-021d-4155-9972-037349415d91" containerName="istio-proxy"
Oct 31 13:37:11.038977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.038848    4428 cpu_manager.go:414] "containerMap: removing container" podUID="60c24e33-d236-4bb6-adcb-3c2160c30b8d" containerName="revoke-keys"
Oct 31 13:37:11.038977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.038852    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c24e33-d236-4bb6-adcb-3c2160c30b8d" containerName="revoke-keys"
Oct 31 13:37:11.039167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.038985    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a514fea-055a-47ce-baa7-cd30cb8a88b4" containerName="istio-proxy"
Oct 31 13:37:11.039167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.038996    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="170b8b28-9f7f-494f-b6e6-86d5af6dcfec" containerName="istio-proxy"
Oct 31 13:37:11.039167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.039003    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6602f73-021d-4155-9972-037349415d91" containerName="istio-proxy"
Oct 31 13:37:11.039167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.039011    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c24e33-d236-4bb6-adcb-3c2160c30b8d" containerName="revoke-keys"
Oct 31 13:37:11.039167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.039019    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b5a82a-0de7-4c45-94a5-b8c046ea9387" containerName="istio-proxy"
Oct 31 13:37:11.039167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.039027    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="04337deb-7a21-4e58-8a41-098d8be2e918" containerName="istio-proxy"
Oct 31 13:37:11.045190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.045172    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"
Oct 31 13:37:11.050228 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.050200    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:37:11.050364 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.050273    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:37:11.052183 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.052162    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"]
Oct 31 13:37:11.078827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.078790    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:11.175007 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.174972    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmg28\" (UniqueName: \"kubernetes.io/projected/79bf625f-03f6-4d45-8227-683270c01391-kube-api-access-kmg28\") pod \"maas-api-revoke-sub-7aa93a88-h8zrs\" (UID: \"79bf625f-03f6-4d45-8227-683270c01391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"
Oct 31 13:37:11.175180 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.175158    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79bf625f-03f6-4d45-8227-683270c01391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7aa93a88-h8zrs\" (UID: \"79bf625f-03f6-4d45-8227-683270c01391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"
Oct 31 13:37:11.175228 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.175213    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79bf625f-03f6-4d45-8227-683270c01391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7aa93a88-h8zrs\" (UID: \"79bf625f-03f6-4d45-8227-683270c01391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"
Oct 31 13:37:11.277302 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.277238    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79bf625f-03f6-4d45-8227-683270c01391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7aa93a88-h8zrs\" (UID: \"79bf625f-03f6-4d45-8227-683270c01391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"
Oct 31 13:37:11.277492 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.277309    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79bf625f-03f6-4d45-8227-683270c01391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7aa93a88-h8zrs\" (UID: \"79bf625f-03f6-4d45-8227-683270c01391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"
Oct 31 13:37:11.277644 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.277552    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmg28\" (UniqueName: \"kubernetes.io/projected/79bf625f-03f6-4d45-8227-683270c01391-kube-api-access-kmg28\") pod \"maas-api-revoke-sub-7aa93a88-h8zrs\" (UID: \"79bf625f-03f6-4d45-8227-683270c01391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"
Oct 31 13:37:11.282503 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.279049    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79bf625f-03f6-4d45-8227-683270c01391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7aa93a88-h8zrs\" (UID: \"79bf625f-03f6-4d45-8227-683270c01391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"
Oct 31 13:37:11.282503 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.281220    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79bf625f-03f6-4d45-8227-683270c01391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7aa93a88-h8zrs\" (UID: \"79bf625f-03f6-4d45-8227-683270c01391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"
Oct 31 13:37:11.292624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.292546    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmg28\" (UniqueName: \"kubernetes.io/projected/79bf625f-03f6-4d45-8227-683270c01391-kube-api-access-kmg28\") pod \"maas-api-revoke-sub-7aa93a88-h8zrs\" (UID: \"79bf625f-03f6-4d45-8227-683270c01391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"
Oct 31 13:37:11.353988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.352900    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"]
Oct 31 13:37:11.363803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.363777    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"
Oct 31 13:37:11.374998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.374970    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"]
Oct 31 13:37:11.417326 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.415954    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"]
Oct 31 13:37:11.424918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.423483    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.444429 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.444397    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:11.464132 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.464097    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"]
Oct 31 13:37:11.491349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.490676    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.491349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.490709    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.491349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.490734    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.491349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.490762    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7b17e1-a025-428d-ad57-a52013e967fc-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.491349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.490823    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.491349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.490961    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.491349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.491000    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.491349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.491022    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.491349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.491042    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmsss\" (UniqueName: \"kubernetes.io/projected/aa7b17e1-a025-428d-ad57-a52013e967fc-kube-api-access-jmsss\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.592373 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.592292    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.592519 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.592437    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.592519 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.592483    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.592519 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.592498    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.592680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.592519    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmsss\" (UniqueName: \"kubernetes.io/projected/aa7b17e1-a025-428d-ad57-a52013e967fc-kube-api-access-jmsss\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.592680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.592568    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.592680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.592598    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.592680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.592625    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.592680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.592663    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7b17e1-a025-428d-ad57-a52013e967fc-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.592998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.592890    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.592998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.592908    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.593412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.593383    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.593591 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.593564    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.593694 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.593576    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7b17e1-a025-428d-ad57-a52013e967fc-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.595148 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.595119    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.595505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.595486    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.607133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.607108    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.607281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.607262    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmsss\" (UniqueName: \"kubernetes.io/projected/aa7b17e1-a025-428d-ad57-a52013e967fc-kube-api-access-jmsss\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.628319 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.628293    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:11.694847 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:37:11.694818    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bf625f_03f6_4d45_8227_683270c01391.slice/crio-a5d671a7f74c562639d289508536d2566bd9557ddec7c63eaca749c10d88d01f WatchSource:0}: Error finding container a5d671a7f74c562639d289508536d2566bd9557ddec7c63eaca749c10d88d01f: Status 404 returned error can't find the container with id a5d671a7f74c562639d289508536d2566bd9557ddec7c63eaca749c10d88d01f
Oct 31 13:37:11.694847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.694841    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"]
Oct 31 13:37:11.777529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.777225    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:11.905988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:11.905960    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:37:12.009571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.009520    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs" event={"ID":"79bf625f-03f6-4d45-8227-683270c01391","Type":"ContainerStarted","Data":"a5d671a7f74c562639d289508536d2566bd9557ddec7c63eaca749c10d88d01f"}
Oct 31 13:37:12.217774 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.217737    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:12.349790 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.349758    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"]
Oct 31 13:37:12.350772 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.350719    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:12.351012 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.350990    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:12.351171 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.351151    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:12.395738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.388773    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:12.574814 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.574781    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"]
Oct 31 13:37:12.589150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.588121    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.615446 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.614499    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f4232c-openshift-default-dockercfg-jrtvl\""
Oct 31 13:37:12.638661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.638627    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgqhk\" (UniqueName: \"kubernetes.io/projected/f0a9f039-bd18-4c3a-88a3-973bc6721086-kube-api-access-pgqhk\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.641552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.639025    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.642343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.642317    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.642580 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.642559    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.642743 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.642721    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.642930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.642909    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.643150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.643128    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.643302 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.643284    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.643472 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.643441    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0a9f039-bd18-4c3a-88a3-973bc6721086-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.646666 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.643295    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"]
Oct 31 13:37:12.717023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.687996    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:12.771192 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.771099    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.771755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.771636    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.788035 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.771967    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0a9f039-bd18-4c3a-88a3-973bc6721086-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.789154 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.789125    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgqhk\" (UniqueName: \"kubernetes.io/projected/f0a9f039-bd18-4c3a-88a3-973bc6721086-kube-api-access-pgqhk\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.791176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.791150    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.791627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.791607    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.854486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.850979    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.854486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.851156    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.854486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.851187    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.854486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.851229    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.854486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.820139    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0a9f039-bd18-4c3a-88a3-973bc6721086-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.854486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.792163    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.872581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.872548    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.875981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.874584    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.884479 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.876450    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.889283 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.889213    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgqhk\" (UniqueName: \"kubernetes.io/projected/f0a9f039-bd18-4c3a-88a3-973bc6721086-kube-api-access-pgqhk\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.910325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.910008    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:12.913505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.913477    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.913733 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.913493    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:12.965166 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:12.965127    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:13.044957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:13.043759    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs" event={"ID":"79bf625f-03f6-4d45-8227-683270c01391","Type":"ContainerStarted","Data":"d2a0d9dfd6a662705da026c249538fc4f0e8a91bf309cd42d23866fbef9348fd"}
Oct 31 13:37:13.048893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:13.048805    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz" event={"ID":"aa7b17e1-a025-428d-ad57-a52013e967fc","Type":"ContainerStarted","Data":"529346514e652c225d3955628b66c6bf35147b0b7be6ef443a998906673d46fd"}
Oct 31 13:37:13.049795 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:13.049454    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz" event={"ID":"aa7b17e1-a025-428d-ad57-a52013e967fc","Type":"ContainerStarted","Data":"f73ce46c6ec44dfea14db74175f5c427ec55b7a2568ad01437f6dec0c96a6ba9"}
Oct 31 13:37:13.117891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:13.114928    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs" podStartSLOduration=2.114906309 podStartE2EDuration="2.114906309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:13.067120005 +0000 UTC m=+2030.465468630" watchObservedRunningTime="2026-10-31 13:37:13.114906309 +0000 UTC m=+2030.513254934"
Oct 31 13:37:13.117891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:13.116604    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz" podStartSLOduration=2.116587563 podStartE2EDuration="2.116587563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:13.104345482 +0000 UTC m=+2030.502694142" watchObservedRunningTime="2026-10-31 13:37:13.116587563 +0000 UTC m=+2030.514936188"
Oct 31 13:37:13.381187 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:13.381155    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"]
Oct 31 13:37:13.381313 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:37:13.381244    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a9f039_bd18_4c3a_88a3_973bc6721086.slice/crio-06f38a239f9cb3834be8b129d720bd76f6700a35253118ce727370f28774c14e WatchSource:0}: Error finding container 06f38a239f9cb3834be8b129d720bd76f6700a35253118ce727370f28774c14e: Status 404 returned error can't find the container with id 06f38a239f9cb3834be8b129d720bd76f6700a35253118ce727370f28774c14e
Oct 31 13:37:13.384349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:13.384317    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:13.384433 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:13.384397    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:13.384470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:13.384442    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:13.412294 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:13.412267    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:13.781942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:13.779256    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:13.783517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:13.783433    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 13:37:13.785500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:13.783525    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz" podUID="aa7b17e1-a025-428d-ad57-a52013e967fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 13:37:13.877992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:13.874148    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:14.078765 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:14.078671    4428 generic.go:356] "Generic (PLEG): container finished" podID="79bf625f-03f6-4d45-8227-683270c01391" containerID="d2a0d9dfd6a662705da026c249538fc4f0e8a91bf309cd42d23866fbef9348fd" exitCode=0
Oct 31 13:37:14.078972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:14.078762    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs" event={"ID":"79bf625f-03f6-4d45-8227-683270c01391","Type":"ContainerDied","Data":"d2a0d9dfd6a662705da026c249538fc4f0e8a91bf309cd42d23866fbef9348fd"}
Oct 31 13:37:14.080608 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:14.080575    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" event={"ID":"f0a9f039-bd18-4c3a-88a3-973bc6721086","Type":"ContainerStarted","Data":"1e26e91b65e5cb9ec7273df8e235af5f5b11d66e27279d7c5205d3e2cf060432"}
Oct 31 13:37:14.080756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:14.080625    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" event={"ID":"f0a9f039-bd18-4c3a-88a3-973bc6721086","Type":"ContainerStarted","Data":"06f38a239f9cb3834be8b129d720bd76f6700a35253118ce727370f28774c14e"}
Oct 31 13:37:14.132026 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:14.131973    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" podStartSLOduration=2.131958381 podStartE2EDuration="2.131958381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:14.128998984 +0000 UTC m=+2031.527347608" watchObservedRunningTime="2026-10-31 13:37:14.131958381 +0000 UTC m=+2031.530307004"
Oct 31 13:37:14.779212 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:14.779174    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 13:37:14.779392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:14.779257    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz" podUID="aa7b17e1-a025-428d-ad57-a52013e967fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 13:37:14.970050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:14.970014    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:14.976653 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:14.976627    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:15.089606 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:15.089172    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:15.090517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:15.090495    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:37:15.241087 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:15.220234    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:15.525742 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:15.525714    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"
Oct 31 13:37:15.580016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:15.579984    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79bf625f-03f6-4d45-8227-683270c01391-kube-api-access-kmg28\" (UniqueName: \"kubernetes.io/projected/79bf625f-03f6-4d45-8227-683270c01391-kube-api-access-kmg28\") pod \"79bf625f-03f6-4d45-8227-683270c01391\" (UID: \"79bf625f-03f6-4d45-8227-683270c01391\") "
Oct 31 13:37:15.580414 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:15.580383    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79bf625f-03f6-4d45-8227-683270c01391-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79bf625f-03f6-4d45-8227-683270c01391-maas-api-openshift-ca\") pod \"79bf625f-03f6-4d45-8227-683270c01391\" (UID: \"79bf625f-03f6-4d45-8227-683270c01391\") "
Oct 31 13:37:15.580800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:15.580774    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79bf625f-03f6-4d45-8227-683270c01391-maas-api-openshift-ca" pod "79bf625f-03f6-4d45-8227-683270c01391" (UID: "79bf625f-03f6-4d45-8227-683270c01391"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:15.581411 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:15.581387    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79bf625f-03f6-4d45-8227-683270c01391-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:15.586459 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:15.586423    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bf625f-03f6-4d45-8227-683270c01391-kube-api-access-kmg28" pod "79bf625f-03f6-4d45-8227-683270c01391" (UID: "79bf625f-03f6-4d45-8227-683270c01391"). InnerVolumeSpecName "kube-api-access-kmg28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:15.683036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:15.683001    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79bf625f-03f6-4d45-8227-683270c01391-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79bf625f-03f6-4d45-8227-683270c01391-maas-api-xks-ca\") pod \"79bf625f-03f6-4d45-8227-683270c01391\" (UID: \"79bf625f-03f6-4d45-8227-683270c01391\") "
Oct 31 13:37:15.683550 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:15.683534    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmg28\" (UniqueName: \"kubernetes.io/projected/79bf625f-03f6-4d45-8227-683270c01391-kube-api-access-kmg28\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:15.685372 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:15.685332    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79bf625f-03f6-4d45-8227-683270c01391-maas-api-xks-ca" pod "79bf625f-03f6-4d45-8227-683270c01391" (UID: "79bf625f-03f6-4d45-8227-683270c01391"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:37:15.779300 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:15.779215    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 13:37:15.779300 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:15.779271    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz" podUID="aa7b17e1-a025-428d-ad57-a52013e967fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 13:37:15.785469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:15.785436    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79bf625f-03f6-4d45-8227-683270c01391-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:16.097774 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:16.097688    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"
Oct 31 13:37:16.097774 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:16.097715    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs" event={"ID":"79bf625f-03f6-4d45-8227-683270c01391","Type":"ContainerDied","Data":"a5d671a7f74c562639d289508536d2566bd9557ddec7c63eaca749c10d88d01f"}
Oct 31 13:37:16.098277 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:16.097792    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5d671a7f74c562639d289508536d2566bd9557ddec7c63eaca749c10d88d01f"
Oct 31 13:37:16.362945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:16.361939    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk" podUID="7b51cfae-67bd-4df7-a6da-802ec754ffe3" containerName="payload-processing" containerID="cri-o://94332cf83a4ad7e82b24fac4c62055fa9faa0a291eb1f4662b0a8fea8cb9e18f" gracePeriod=25
Oct 31 13:37:16.376459 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:16.376240    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p" podUID="b51ddd17-4a94-468b-9ee7-c2f0de9b4528" containerName="payload-pre-processing" containerID="cri-o://3877cb11cf92ccf40aece332338eb9cb55aa68c4b487c3decd13f2282cae6004" gracePeriod=25
Oct 31 13:37:16.789889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:16.787408    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 13:37:16.789889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:16.787477    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz" podUID="aa7b17e1-a025-428d-ad57-a52013e967fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 13:37:16.995720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:16.995690    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"
Oct 31 13:37:17.091570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.089813    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b51ddd17-4a94-468b-9ee7-c2f0de9b4528-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51ddd17-4a94-468b-9ee7-c2f0de9b4528-plugins-config-volume\") pod \"b51ddd17-4a94-468b-9ee7-c2f0de9b4528\" (UID: \"b51ddd17-4a94-468b-9ee7-c2f0de9b4528\") "
Oct 31 13:37:17.091570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.089906    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b51ddd17-4a94-468b-9ee7-c2f0de9b4528-kube-api-access-24k24\" (UniqueName: \"kubernetes.io/projected/b51ddd17-4a94-468b-9ee7-c2f0de9b4528-kube-api-access-24k24\") pod \"b51ddd17-4a94-468b-9ee7-c2f0de9b4528\" (UID: \"b51ddd17-4a94-468b-9ee7-c2f0de9b4528\") "
Oct 31 13:37:17.091570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.091479    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b51ddd17-4a94-468b-9ee7-c2f0de9b4528-plugins-config-volume" pod "b51ddd17-4a94-468b-9ee7-c2f0de9b4528" (UID: "b51ddd17-4a94-468b-9ee7-c2f0de9b4528"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:17.095632 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.095602    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51ddd17-4a94-468b-9ee7-c2f0de9b4528-kube-api-access-24k24" pod "b51ddd17-4a94-468b-9ee7-c2f0de9b4528" (UID: "b51ddd17-4a94-468b-9ee7-c2f0de9b4528"). InnerVolumeSpecName "kube-api-access-24k24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:17.138958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.138926    4428 generic.go:356] "Generic (PLEG): container finished" podID="b51ddd17-4a94-468b-9ee7-c2f0de9b4528" containerID="3877cb11cf92ccf40aece332338eb9cb55aa68c4b487c3decd13f2282cae6004" exitCode=0
Oct 31 13:37:17.139329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.139013    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p" event={"ID":"b51ddd17-4a94-468b-9ee7-c2f0de9b4528","Type":"ContainerDied","Data":"3877cb11cf92ccf40aece332338eb9cb55aa68c4b487c3decd13f2282cae6004"}
Oct 31 13:37:17.139329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.139025    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"
Oct 31 13:37:17.139329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.139054    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p" event={"ID":"b51ddd17-4a94-468b-9ee7-c2f0de9b4528","Type":"ContainerDied","Data":"3843a5ca6a3b79e436863ead75cbb0209e9fda715dfd387bd8a75626712331fe"}
Oct 31 13:37:17.139329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.139076    4428 scope.go:122] "RemoveContainer" containerID="3877cb11cf92ccf40aece332338eb9cb55aa68c4b487c3decd13f2282cae6004"
Oct 31 13:37:17.140934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.140908    4428 generic.go:356] "Generic (PLEG): container finished" podID="7b51cfae-67bd-4df7-a6da-802ec754ffe3" containerID="94332cf83a4ad7e82b24fac4c62055fa9faa0a291eb1f4662b0a8fea8cb9e18f" exitCode=0
Oct 31 13:37:17.141057 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.141029    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk" event={"ID":"7b51cfae-67bd-4df7-a6da-802ec754ffe3","Type":"ContainerDied","Data":"94332cf83a4ad7e82b24fac4c62055fa9faa0a291eb1f4662b0a8fea8cb9e18f"}
Oct 31 13:37:17.142741 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.142713    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:17.186481 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.186457    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"
Oct 31 13:37:17.187717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.186997    4428 scope.go:122] "RemoveContainer" containerID="3877cb11cf92ccf40aece332338eb9cb55aa68c4b487c3decd13f2282cae6004"
Oct 31 13:37:17.187717 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:37:17.187326    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3877cb11cf92ccf40aece332338eb9cb55aa68c4b487c3decd13f2282cae6004\": container with ID starting with 3877cb11cf92ccf40aece332338eb9cb55aa68c4b487c3decd13f2282cae6004 not found: ID does not exist" containerID="3877cb11cf92ccf40aece332338eb9cb55aa68c4b487c3decd13f2282cae6004"
Oct 31 13:37:17.187717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.187361    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3877cb11cf92ccf40aece332338eb9cb55aa68c4b487c3decd13f2282cae6004"} err="failed to get container status \"3877cb11cf92ccf40aece332338eb9cb55aa68c4b487c3decd13f2282cae6004\": rpc error: code = NotFound desc = could not find container \"3877cb11cf92ccf40aece332338eb9cb55aa68c4b487c3decd13f2282cae6004\": container with ID starting with 3877cb11cf92ccf40aece332338eb9cb55aa68c4b487c3decd13f2282cae6004 not found: ID does not exist"
Oct 31 13:37:17.192241 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.192202    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51ddd17-4a94-468b-9ee7-c2f0de9b4528-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:17.192241 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.192233    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24k24\" (UniqueName: \"kubernetes.io/projected/b51ddd17-4a94-468b-9ee7-c2f0de9b4528-kube-api-access-24k24\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:17.321900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.299678    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b51cfae-67bd-4df7-a6da-802ec754ffe3-kube-api-access-r84bq\" (UniqueName: \"kubernetes.io/projected/7b51cfae-67bd-4df7-a6da-802ec754ffe3-kube-api-access-r84bq\") pod \"7b51cfae-67bd-4df7-a6da-802ec754ffe3\" (UID: \"7b51cfae-67bd-4df7-a6da-802ec754ffe3\") "
Oct 31 13:37:17.321900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.300082    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b51cfae-67bd-4df7-a6da-802ec754ffe3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b51cfae-67bd-4df7-a6da-802ec754ffe3-plugins-config-volume\") pod \"7b51cfae-67bd-4df7-a6da-802ec754ffe3\" (UID: \"7b51cfae-67bd-4df7-a6da-802ec754ffe3\") "
Oct 31 13:37:17.490835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.384388    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:17.490835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.477795    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b51cfae-67bd-4df7-a6da-802ec754ffe3-plugins-config-volume" pod "7b51cfae-67bd-4df7-a6da-802ec754ffe3" (UID: "7b51cfae-67bd-4df7-a6da-802ec754ffe3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:17.545893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.515326    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b51cfae-67bd-4df7-a6da-802ec754ffe3-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:17.562894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.553092    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b51cfae-67bd-4df7-a6da-802ec754ffe3-kube-api-access-r84bq" pod "7b51cfae-67bd-4df7-a6da-802ec754ffe3" (UID: "7b51cfae-67bd-4df7-a6da-802ec754ffe3"). InnerVolumeSpecName "kube-api-access-r84bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:17.627627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.627512    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r84bq\" (UniqueName: \"kubernetes.io/projected/7b51cfae-67bd-4df7-a6da-802ec754ffe3-kube-api-access-r84bq\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:17.640363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.640290    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:17.640553 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.640535    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"]
Oct 31 13:37:17.643116 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.641776    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6aa924-66d4f47767-ghf8p"]
Oct 31 13:37:17.778458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.778421    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 13:37:17.778631 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:17.778468    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz" podUID="aa7b17e1-a025-428d-ad57-a52013e967fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 13:37:18.179283 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.179254    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"
Oct 31 13:37:18.179283 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.179280    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk" event={"ID":"7b51cfae-67bd-4df7-a6da-802ec754ffe3","Type":"ContainerDied","Data":"91c8e021cb302f40fb91fedd9d07b5b36c96aef44a692612a795cb6ade80d4bc"}
Oct 31 13:37:18.180056 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.179333    4428 scope.go:122] "RemoveContainer" containerID="94332cf83a4ad7e82b24fac4c62055fa9faa0a291eb1f4662b0a8fea8cb9e18f"
Oct 31 13:37:18.213568 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.213546    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"]
Oct 31 13:37:18.218802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.218768    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6aa924-775b66d469-82hlk"]
Oct 31 13:37:18.612614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.612591    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:18.636480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.636453    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"]
Oct 31 13:37:18.637256 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.637237    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b51ddd17-4a94-468b-9ee7-c2f0de9b4528" containerName="payload-pre-processing"
Oct 31 13:37:18.637256 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.637256    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b51ddd17-4a94-468b-9ee7-c2f0de9b4528" containerName="payload-pre-processing"
Oct 31 13:37:18.637392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.637290    4428 cpu_manager.go:414] "containerMap: removing container" podUID="7b51cfae-67bd-4df7-a6da-802ec754ffe3" containerName="payload-processing"
Oct 31 13:37:18.637392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.637295    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b51cfae-67bd-4df7-a6da-802ec754ffe3" containerName="payload-processing"
Oct 31 13:37:18.637392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.637307    4428 cpu_manager.go:414] "containerMap: removing container" podUID="79bf625f-03f6-4d45-8227-683270c01391" containerName="revoke-keys"
Oct 31 13:37:18.637392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.637311    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79bf625f-03f6-4d45-8227-683270c01391" containerName="revoke-keys"
Oct 31 13:37:18.637524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.637419    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b51ddd17-4a94-468b-9ee7-c2f0de9b4528" containerName="payload-pre-processing"
Oct 31 13:37:18.637524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.637436    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b51cfae-67bd-4df7-a6da-802ec754ffe3" containerName="payload-processing"
Oct 31 13:37:18.637524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.637442    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="79bf625f-03f6-4d45-8227-683270c01391" containerName="revoke-keys"
Oct 31 13:37:18.642590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.642507    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"
Oct 31 13:37:18.653903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.653231    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f4232c\""
Oct 31 13:37:18.655573 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.654730    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f4232c-dockercfg-ml82c\""
Oct 31 13:37:18.682406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.682372    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"]
Oct 31 13:37:18.703719 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.703687    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"]
Oct 31 13:37:18.719382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.717790    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"]
Oct 31 13:37:18.719382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.717940    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"
Oct 31 13:37:18.791219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.786581    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fb736b6-9425-445b-85bb-be3f5a9598d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj\" (UID: \"1fb736b6-9425-445b-85bb-be3f5a9598d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"
Oct 31 13:37:18.791219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.786714    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4gzn\" (UniqueName: \"kubernetes.io/projected/1fb736b6-9425-445b-85bb-be3f5a9598d1-kube-api-access-w4gzn\") pod \"payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj\" (UID: \"1fb736b6-9425-445b-85bb-be3f5a9598d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"
Oct 31 13:37:18.792037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.791992    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 13:37:18.792257 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.792064    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz" podUID="aa7b17e1-a025-428d-ad57-a52013e967fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 13:37:18.807049 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.807011    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:18.861967 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.858491    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b51cfae-67bd-4df7-a6da-802ec754ffe3" path="/var/lib/kubelet/pods/7b51cfae-67bd-4df7-a6da-802ec754ffe3/volumes"
Oct 31 13:37:18.861967 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:18.859232    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b51ddd17-4a94-468b-9ee7-c2f0de9b4528" path="/var/lib/kubelet/pods/b51ddd17-4a94-468b-9ee7-c2f0de9b4528/volumes"
Oct 31 13:37:18.900106 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:37:18.900008    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:37:19.044037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.038488    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:19.095916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.093191    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c659c\" (UniqueName: \"kubernetes.io/projected/67b60c41-c1fe-4112-88ea-423d2dc01997-kube-api-access-c659c\") pod \"payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8\" (UID: \"67b60c41-c1fe-4112-88ea-423d2dc01997\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"
Oct 31 13:37:19.095916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.093328    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b60c41-c1fe-4112-88ea-423d2dc01997-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8\" (UID: \"67b60c41-c1fe-4112-88ea-423d2dc01997\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"
Oct 31 13:37:19.095916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.093391    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fb736b6-9425-445b-85bb-be3f5a9598d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj\" (UID: \"1fb736b6-9425-445b-85bb-be3f5a9598d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"
Oct 31 13:37:19.095916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.093488    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4gzn\" (UniqueName: \"kubernetes.io/projected/1fb736b6-9425-445b-85bb-be3f5a9598d1-kube-api-access-w4gzn\") pod \"payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj\" (UID: \"1fb736b6-9425-445b-85bb-be3f5a9598d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"
Oct 31 13:37:19.150399 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.150314    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fb736b6-9425-445b-85bb-be3f5a9598d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj\" (UID: \"1fb736b6-9425-445b-85bb-be3f5a9598d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"
Oct 31 13:37:19.178266 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.178235    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4gzn\" (UniqueName: \"kubernetes.io/projected/1fb736b6-9425-445b-85bb-be3f5a9598d1-kube-api-access-w4gzn\") pod \"payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj\" (UID: \"1fb736b6-9425-445b-85bb-be3f5a9598d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"
Oct 31 13:37:19.196761 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.196723    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c659c\" (UniqueName: \"kubernetes.io/projected/67b60c41-c1fe-4112-88ea-423d2dc01997-kube-api-access-c659c\") pod \"payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8\" (UID: \"67b60c41-c1fe-4112-88ea-423d2dc01997\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"
Oct 31 13:37:19.197472 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.197446    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b60c41-c1fe-4112-88ea-423d2dc01997-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8\" (UID: \"67b60c41-c1fe-4112-88ea-423d2dc01997\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"
Oct 31 13:37:19.198754 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.198729    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b60c41-c1fe-4112-88ea-423d2dc01997-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8\" (UID: \"67b60c41-c1fe-4112-88ea-423d2dc01997\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"
Oct 31 13:37:19.215638 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.215368    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c659c\" (UniqueName: \"kubernetes.io/projected/67b60c41-c1fe-4112-88ea-423d2dc01997-kube-api-access-c659c\") pod \"payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8\" (UID: \"67b60c41-c1fe-4112-88ea-423d2dc01997\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"
Oct 31 13:37:19.298654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.298621    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"
Oct 31 13:37:19.309510 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.307927    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"]
Oct 31 13:37:19.321457 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.321413    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"]
Oct 31 13:37:19.321611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.321539    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.327724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.326995    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fb50fa-openshift-default-dockercfg-7tzdl\""
Oct 31 13:37:19.395256 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.392721    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"
Oct 31 13:37:19.415949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.413019    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:19.537822 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.537783    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnnln\" (UniqueName: \"kubernetes.io/projected/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-kube-api-access-fnnln\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.538142 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.538118    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-workload-certs\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.538296 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.538278    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-envoy\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.538496 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.538479    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-credential-socket\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.538615 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.538600    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-workload-socket\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.538740 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.538727    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-podinfo\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.538985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.538967    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-token\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.539175 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.539158    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istiod-ca-cert\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.539312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.539296    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-data\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.592230 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.592147    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:19.690946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.685984    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-podinfo\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.690946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.686244    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-token\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.690946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.686347    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istiod-ca-cert\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.690946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.686402    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-data\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.690946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.686458    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnnln\" (UniqueName: \"kubernetes.io/projected/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-kube-api-access-fnnln\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.690946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.686518    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-workload-certs\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.690946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.686579    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-envoy\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.690946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.686690    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-credential-socket\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.690946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.686738    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-workload-socket\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.690946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.687333    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-workload-socket\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.698886 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.698525    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-data\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.712909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.699722    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-workload-certs\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.712909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.699988    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-credential-socket\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.714360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.714322    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istiod-ca-cert\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.789502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.785264    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-token\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:19.857681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.855026    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 13:37:19.857681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.855090    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz" podUID="aa7b17e1-a025-428d-ad57-a52013e967fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 13:37:19.857681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:19.856575    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-podinfo\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:20.013314 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:20.013279    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnnln\" (UniqueName: \"kubernetes.io/projected/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-kube-api-access-fnnln\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:20.052725 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:20.017757    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-envoy\") pod \"e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") " pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:20.052725 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:20.042013    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:20.159341 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:20.159310    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:20.245332 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:20.243754    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"]
Oct 31 13:37:20.256926 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:37:20.253721    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb736b6_9425_445b_85bb_be3f5a9598d1.slice/crio-f7444b98e51648e278f0f629ccaab8b2d91b2a179bddb8a283232c27225ba144 WatchSource:0}: Error finding container f7444b98e51648e278f0f629ccaab8b2d91b2a179bddb8a283232c27225ba144: Status 404 returned error can't find the container with id f7444b98e51648e278f0f629ccaab8b2d91b2a179bddb8a283232c27225ba144
Oct 31 13:37:20.296908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:20.294031    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"]
Oct 31 13:37:20.300568 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:37:20.300536    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b60c41_c1fe_4112_88ea_423d2dc01997.slice/crio-f75339b7319e982149b2674442115c02906934b42e2e116c41c259db3360e389 WatchSource:0}: Error finding container f75339b7319e982149b2674442115c02906934b42e2e116c41c259db3360e389: Status 404 returned error can't find the container with id f75339b7319e982149b2674442115c02906934b42e2e116c41c259db3360e389
Oct 31 13:37:20.471759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:20.471727    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"]
Oct 31 13:37:20.472203 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:37:20.472173    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc8b218_3e92_4c29_b42f_9eb3dbdd2d74.slice/crio-b7ded35ba08684ca3042a5d50715c15f4618c214e371d5028d2e1969979f8b18 WatchSource:0}: Error finding container b7ded35ba08684ca3042a5d50715c15f4618c214e371d5028d2e1969979f8b18: Status 404 returned error can't find the container with id b7ded35ba08684ca3042a5d50715c15f4618c214e371d5028d2e1969979f8b18
Oct 31 13:37:20.475075 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:20.475043    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:20.475168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:20.475116    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:20.475168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:20.475159    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:20.783593 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:20.783517    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:20.783922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:20.783893    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:20.784657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:20.784640    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:37:21.226177 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:21.224691    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8" event={"ID":"67b60c41-c1fe-4112-88ea-423d2dc01997","Type":"ContainerStarted","Data":"e61c40fec420930433b207f0f2357e153b08c6cf356d9d9e2e51a2eaa4d019bb"}
Oct 31 13:37:21.226177 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:21.224756    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8" event={"ID":"67b60c41-c1fe-4112-88ea-423d2dc01997","Type":"ContainerStarted","Data":"f75339b7319e982149b2674442115c02906934b42e2e116c41c259db3360e389"}
Oct 31 13:37:21.226177 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:21.224797    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"
Oct 31 13:37:21.237672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:21.237630    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" event={"ID":"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74","Type":"ContainerStarted","Data":"571c05265fb8792d5e55bf6d8a08b690d0efb1dfa84fafa7319e6ce33250ec7a"}
Oct 31 13:37:21.237848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:21.237732    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" event={"ID":"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74","Type":"ContainerStarted","Data":"b7ded35ba08684ca3042a5d50715c15f4618c214e371d5028d2e1969979f8b18"}
Oct 31 13:37:21.239700 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:21.239673    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj" event={"ID":"1fb736b6-9425-445b-85bb-be3f5a9598d1","Type":"ContainerStarted","Data":"b4e65dadf04e7eb0b0103a88896d02ebdd5495eaec350a62865542013c8f9fa2"}
Oct 31 13:37:21.239898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:21.239724    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj" event={"ID":"1fb736b6-9425-445b-85bb-be3f5a9598d1","Type":"ContainerStarted","Data":"f7444b98e51648e278f0f629ccaab8b2d91b2a179bddb8a283232c27225ba144"}
Oct 31 13:37:21.239898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:21.239800    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"
Oct 31 13:37:21.247743 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:21.247696    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8" podStartSLOduration=3.247684225 podStartE2EDuration="3.247684225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:21.246127363 +0000 UTC m=+2038.644475990" watchObservedRunningTime="2026-10-31 13:37:21.247684225 +0000 UTC m=+2038.646032848"
Oct 31 13:37:21.276373 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:21.276313    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" podStartSLOduration=2.276293789 podStartE2EDuration="2.276293789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:21.27331588 +0000 UTC m=+2038.671664503" watchObservedRunningTime="2026-10-31 13:37:21.276293789 +0000 UTC m=+2038.674642414"
Oct 31 13:37:21.297533 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:21.297486    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj" podStartSLOduration=3.297474865 podStartE2EDuration="3.297474865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:21.296960681 +0000 UTC m=+2038.695309306" watchObservedRunningTime="2026-10-31 13:37:21.297474865 +0000 UTC m=+2038.695823489"
Oct 31 13:37:21.811685 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:21.811627    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:22.092174 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.092086    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:22.160102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.160076    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:22.166389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.166362    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:22.182769 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.182736    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"]
Oct 31 13:37:22.191824 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.191800    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.198527 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.198504    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-62778a-openshift-default-dockercfg-t6zwr\""
Oct 31 13:37:22.199319 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.199296    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"]
Oct 31 13:37:22.251949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.251917    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:22.252659 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.252637    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:37:22.284895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.274183    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6517163-656a-4391-a298-4c187010beae-istio-token\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.284895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.274237    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss2lt\" (UniqueName: \"kubernetes.io/projected/e6517163-656a-4391-a298-4c187010beae-kube-api-access-ss2lt\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.284895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.274299    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6517163-656a-4391-a298-4c187010beae-istiod-ca-cert\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.284895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.274359    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-istio-envoy\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.284895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.274416    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-credential-socket\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.284895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.274483    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-workload-certs\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.284895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.274530    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-istio-data\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.284895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.274649    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-workload-socket\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.284895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.274737    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6517163-656a-4391-a298-4c187010beae-istio-podinfo\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.300913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.296237    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:22.383024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.376567    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-workload-certs\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.383024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.376662    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-istio-data\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.383024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.376845    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-workload-socket\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.404185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.397989    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-istio-data\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.513587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.507435    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-workload-socket\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.547167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.547010    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-workload-certs\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.547649 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.547630    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6517163-656a-4391-a298-4c187010beae-istio-podinfo\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.563892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.562296    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6517163-656a-4391-a298-4c187010beae-istio-token\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.563892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.562541    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss2lt\" (UniqueName: \"kubernetes.io/projected/e6517163-656a-4391-a298-4c187010beae-kube-api-access-ss2lt\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.563892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.562713    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6517163-656a-4391-a298-4c187010beae-istiod-ca-cert\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.563892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.563237    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-istio-envoy\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.594497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.568586    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-credential-socket\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.594497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.563995    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6517163-656a-4391-a298-4c187010beae-istiod-ca-cert\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.598711 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.598677    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-credential-socket\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.628358 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.628303    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6517163-656a-4391-a298-4c187010beae-istio-podinfo\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.634941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.629057    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-istio-envoy\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.663897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.656042    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6517163-656a-4391-a298-4c187010beae-istio-token\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.686766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.686735    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss2lt\" (UniqueName: \"kubernetes.io/projected/e6517163-656a-4391-a298-4c187010beae-kube-api-access-ss2lt\") pod \"e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") " pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.812394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.812363    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:22.960989 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:22.960203    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:23.109753 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:37:23.109722    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6517163_656a_4391_a298_4c187010beae.slice/crio-99678ff9c2a27787ad66b21b3a5b10ab7acd68290abd52aaffbf7903900b0216 WatchSource:0}: Error finding container 99678ff9c2a27787ad66b21b3a5b10ab7acd68290abd52aaffbf7903900b0216: Status 404 returned error can't find the container with id 99678ff9c2a27787ad66b21b3a5b10ab7acd68290abd52aaffbf7903900b0216
Oct 31 13:37:23.111226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:23.111195    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"]
Oct 31 13:37:23.112827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:23.112797    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:23.112912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:23.112859    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:23.112957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:23.112936    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:23.263981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:23.263901    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" event={"ID":"e6517163-656a-4391-a298-4c187010beae","Type":"ContainerStarted","Data":"ad4686d66ca13445f4e4dff2af2ef7904296be88a65e415bef969e3a3d22d10c"}
Oct 31 13:37:23.263981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:23.263961    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" event={"ID":"e6517163-656a-4391-a298-4c187010beae","Type":"ContainerStarted","Data":"99678ff9c2a27787ad66b21b3a5b10ab7acd68290abd52aaffbf7903900b0216"}
Oct 31 13:37:23.292448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:23.292379    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" podStartSLOduration=1.292360333 podStartE2EDuration="1.292360333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:23.28792802 +0000 UTC m=+2040.686276646" watchObservedRunningTime="2026-10-31 13:37:23.292360333 +0000 UTC m=+2040.690708963"
Oct 31 13:37:23.713158 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:23.713124    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:23.812619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:23.812588    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:23.891628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:23.891595    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:24.125154 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:24.125070    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:24.819306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:24.819272    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:24.860916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:24.854002    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:25.210785 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.210753    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:25.245804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.245778    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"]
Oct 31 13:37:25.251326 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.251301    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"
Oct 31 13:37:25.256424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.256403    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fb50fa\""
Oct 31 13:37:25.256787 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.256769    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fb50fa-dockercfg-w5n4c\""
Oct 31 13:37:25.261976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.261724    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"]
Oct 31 13:37:25.267803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.267576    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"
Oct 31 13:37:25.268282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.268262    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"]
Oct 31 13:37:25.290840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.290818    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:25.292146 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.292122    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:37:25.293692 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.293633    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"]
Oct 31 13:37:25.574641 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.565502    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:25.605219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.603052    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d14076-4049-4177-9a99-738b8d8e808b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg\" (UID: \"e5d14076-4049-4177-9a99-738b8d8e808b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"
Oct 31 13:37:25.605511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.605484    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1769894b-3274-4280-b0e2-a7056d77a96a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n\" (UID: \"1769894b-3274-4280-b0e2-a7056d77a96a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"
Oct 31 13:37:25.605722 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.605705    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc2xf\" (UniqueName: \"kubernetes.io/projected/e5d14076-4049-4177-9a99-738b8d8e808b-kube-api-access-rc2xf\") pod \"payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg\" (UID: \"e5d14076-4049-4177-9a99-738b8d8e808b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"
Oct 31 13:37:25.606302 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.606276    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxmvl\" (UniqueName: \"kubernetes.io/projected/1769894b-3274-4280-b0e2-a7056d77a96a-kube-api-access-rxmvl\") pod \"payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n\" (UID: \"1769894b-3274-4280-b0e2-a7056d77a96a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"
Oct 31 13:37:25.722235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.714138    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d14076-4049-4177-9a99-738b8d8e808b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg\" (UID: \"e5d14076-4049-4177-9a99-738b8d8e808b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"
Oct 31 13:37:25.722235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.714193    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1769894b-3274-4280-b0e2-a7056d77a96a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n\" (UID: \"1769894b-3274-4280-b0e2-a7056d77a96a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"
Oct 31 13:37:25.722235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.714252    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc2xf\" (UniqueName: \"kubernetes.io/projected/e5d14076-4049-4177-9a99-738b8d8e808b-kube-api-access-rc2xf\") pod \"payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg\" (UID: \"e5d14076-4049-4177-9a99-738b8d8e808b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"
Oct 31 13:37:25.722235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.714575    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxmvl\" (UniqueName: \"kubernetes.io/projected/1769894b-3274-4280-b0e2-a7056d77a96a-kube-api-access-rxmvl\") pod \"payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n\" (UID: \"1769894b-3274-4280-b0e2-a7056d77a96a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"
Oct 31 13:37:25.722235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.716309    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d14076-4049-4177-9a99-738b8d8e808b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg\" (UID: \"e5d14076-4049-4177-9a99-738b8d8e808b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"
Oct 31 13:37:25.737175 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.729352    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1769894b-3274-4280-b0e2-a7056d77a96a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n\" (UID: \"1769894b-3274-4280-b0e2-a7056d77a96a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"
Oct 31 13:37:25.759961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.759578    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxmvl\" (UniqueName: \"kubernetes.io/projected/1769894b-3274-4280-b0e2-a7056d77a96a-kube-api-access-rxmvl\") pod \"payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n\" (UID: \"1769894b-3274-4280-b0e2-a7056d77a96a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"
Oct 31 13:37:25.776604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.776569    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc2xf\" (UniqueName: \"kubernetes.io/projected/e5d14076-4049-4177-9a99-738b8d8e808b-kube-api-access-rc2xf\") pod \"payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg\" (UID: \"e5d14076-4049-4177-9a99-738b8d8e808b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"
Oct 31 13:37:25.813006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.808524    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:25.889130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.889047    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"
Oct 31 13:37:25.901235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.901205    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"
Oct 31 13:37:25.974673 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:25.974634    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:26.203785 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:37:26.203757    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d14076_4049_4177_9a99_738b8d8e808b.slice/crio-73e8947b6f88569bfa547c317bf02d3e4b31c2c7d31de5b1e75d6213878d632a WatchSource:0}: Error finding container 73e8947b6f88569bfa547c317bf02d3e4b31c2c7d31de5b1e75d6213878d632a: Status 404 returned error can't find the container with id 73e8947b6f88569bfa547c317bf02d3e4b31c2c7d31de5b1e75d6213878d632a
Oct 31 13:37:26.204013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:26.203992    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"]
Oct 31 13:37:26.206662 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:37:26.206638    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1769894b_3274_4280_b0e2_a7056d77a96a.slice/crio-cc856ef7b639192c95bd0fab22d95f43029e2e3d5afdc8f1d33e8b7742d6e6ca WatchSource:0}: Error finding container cc856ef7b639192c95bd0fab22d95f43029e2e3d5afdc8f1d33e8b7742d6e6ca: Status 404 returned error can't find the container with id cc856ef7b639192c95bd0fab22d95f43029e2e3d5afdc8f1d33e8b7742d6e6ca
Oct 31 13:37:26.209433 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:26.209406    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"]
Oct 31 13:37:26.300740 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:26.300700    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg" event={"ID":"e5d14076-4049-4177-9a99-738b8d8e808b","Type":"ContainerStarted","Data":"dfbd850fd86091acd667d73c628781f3e3de84147b5fe30ebdf4e6a57c239648"}
Oct 31 13:37:26.300909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:26.300769    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg" event={"ID":"e5d14076-4049-4177-9a99-738b8d8e808b","Type":"ContainerStarted","Data":"73e8947b6f88569bfa547c317bf02d3e4b31c2c7d31de5b1e75d6213878d632a"}
Oct 31 13:37:26.300909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:26.300800    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"
Oct 31 13:37:26.303664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:26.303637    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n" event={"ID":"1769894b-3274-4280-b0e2-a7056d77a96a","Type":"ContainerStarted","Data":"d4fe255e47338f623d48d69d838488b47d653bb8499600f7cc8047a29e178032"}
Oct 31 13:37:26.303760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:26.303684    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n" event={"ID":"1769894b-3274-4280-b0e2-a7056d77a96a","Type":"ContainerStarted","Data":"cc856ef7b639192c95bd0fab22d95f43029e2e3d5afdc8f1d33e8b7742d6e6ca"}
Oct 31 13:37:26.345795 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:26.345747    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg" podStartSLOduration=1.345733605 podStartE2EDuration="1.345733605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:26.344536808 +0000 UTC m=+2043.742885433" watchObservedRunningTime="2026-10-31 13:37:26.345733605 +0000 UTC m=+2043.744082229"
Oct 31 13:37:26.372523 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:26.372201    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n" podStartSLOduration=1.3721853450000001 podStartE2EDuration="1.372185345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:26.371287522 +0000 UTC m=+2043.769636145" watchObservedRunningTime="2026-10-31 13:37:26.372185345 +0000 UTC m=+2043.770533970"
Oct 31 13:37:26.566007 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:26.565971    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:27.080237 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.080206    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:27.102741 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.102713    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"]
Oct 31 13:37:27.107596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.107581    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"
Oct 31 13:37:27.115759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.115729    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-62778a-dockercfg-k4gd5\""
Oct 31 13:37:27.119212 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.119191    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-62778a\""
Oct 31 13:37:27.125876 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.125833    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"]
Oct 31 13:37:27.149624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.149590    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"]
Oct 31 13:37:27.154886 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.154846    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"
Oct 31 13:37:27.168836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.168809    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"]
Oct 31 13:37:27.183087 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.179826    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fl62\" (UniqueName: \"kubernetes.io/projected/8c73f7d7-3bac-40e8-b386-3742d5a73225-kube-api-access-2fl62\") pod \"payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7\" (UID: \"8c73f7d7-3bac-40e8-b386-3742d5a73225\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"
Oct 31 13:37:27.183087 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.179956    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c73f7d7-3bac-40e8-b386-3742d5a73225-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7\" (UID: \"8c73f7d7-3bac-40e8-b386-3742d5a73225\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"
Oct 31 13:37:27.331029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.319410    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:27.374893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.360932    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fl62\" (UniqueName: \"kubernetes.io/projected/8c73f7d7-3bac-40e8-b386-3742d5a73225-kube-api-access-2fl62\") pod \"payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7\" (UID: \"8c73f7d7-3bac-40e8-b386-3742d5a73225\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"
Oct 31 13:37:27.374893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.361055    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c73f7d7-3bac-40e8-b386-3742d5a73225-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7\" (UID: \"8c73f7d7-3bac-40e8-b386-3742d5a73225\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"
Oct 31 13:37:27.374893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.362289    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c73f7d7-3bac-40e8-b386-3742d5a73225-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7\" (UID: \"8c73f7d7-3bac-40e8-b386-3742d5a73225\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"
Oct 31 13:37:27.477896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.471262    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f\" (UID: \"9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"
Oct 31 13:37:27.477896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.471385    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrnlq\" (UniqueName: \"kubernetes.io/projected/9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c-kube-api-access-hrnlq\") pod \"payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f\" (UID: \"9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"
Oct 31 13:37:27.504220 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.504124    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"
Oct 31 13:37:27.518891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.515673    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fl62\" (UniqueName: \"kubernetes.io/projected/8c73f7d7-3bac-40e8-b386-3742d5a73225-kube-api-access-2fl62\") pod \"payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7\" (UID: \"8c73f7d7-3bac-40e8-b386-3742d5a73225\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"
Oct 31 13:37:27.557176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.557099    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:27.576335 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.576304    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f\" (UID: \"9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"
Oct 31 13:37:27.577046 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.577009    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrnlq\" (UniqueName: \"kubernetes.io/projected/9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c-kube-api-access-hrnlq\") pod \"payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f\" (UID: \"9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"
Oct 31 13:37:27.578935 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.578912    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f\" (UID: \"9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"
Oct 31 13:37:27.619501 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.618452    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrnlq\" (UniqueName: \"kubernetes.io/projected/9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c-kube-api-access-hrnlq\") pod \"payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f\" (UID: \"9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"
Oct 31 13:37:27.755891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.750210    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"
Oct 31 13:37:27.781918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:27.781081    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"
Oct 31 13:37:28.112045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:28.108403    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:28.340891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:28.325604    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:28.499897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:28.498344    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"]
Oct 31 13:37:28.745279 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:28.745249    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-580f24-8t8pm"]
Oct 31 13:37:28.779256 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:28.779216    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d59eb6-d9aa-4795-8573-b08fb31d11b1" path="/var/lib/kubelet/pods/b4d59eb6-d9aa-4795-8573-b08fb31d11b1/volumes"
Oct 31 13:37:28.779634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:28.779613    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"]
Oct 31 13:37:28.779732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:28.779671    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"]
Oct 31 13:37:28.851607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:28.851581    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:29.205139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:29.201374    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:29.781926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:29.781883    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f" event={"ID":"9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c","Type":"ContainerStarted","Data":"9a13e70784f0e73ac0cf3c3617cc9881af581ba1afca128a128580378b8d54fe"}
Oct 31 13:37:29.782159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:29.781956    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f" event={"ID":"9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c","Type":"ContainerStarted","Data":"bc24aa524eaf6f5cba0adc11f548b7e7e50b137673153ae8cdff1c8a3afe9145"}
Oct 31 13:37:29.782159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:29.782027    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"
Oct 31 13:37:29.783936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:29.783905    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7" event={"ID":"8c73f7d7-3bac-40e8-b386-3742d5a73225","Type":"ContainerStarted","Data":"d70b97a5f61954d3edfbd4a0196f8032ee138f06a4290d8b8793fcc42836bddb"}
Oct 31 13:37:29.784082 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:29.783964    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7" event={"ID":"8c73f7d7-3bac-40e8-b386-3742d5a73225","Type":"ContainerStarted","Data":"c80578ea917c04c320dda3388e6555327339cf6f79a59f44dbc80e3d8b3bc1c7"}
Oct 31 13:37:29.784082 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:29.783995    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"
Oct 31 13:37:29.807939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:29.807881    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f" podStartSLOduration=2.807849871 podStartE2EDuration="2.807849871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:29.801894843 +0000 UTC m=+2047.200243468" watchObservedRunningTime="2026-10-31 13:37:29.807849871 +0000 UTC m=+2047.206198494"
Oct 31 13:37:29.833825 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:29.833762    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7" podStartSLOduration=2.833746459 podStartE2EDuration="2.833746459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:29.831240877 +0000 UTC m=+2047.229589502" watchObservedRunningTime="2026-10-31 13:37:29.833746459 +0000 UTC m=+2047.232095082"
Oct 31 13:37:30.572888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:30.563679    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:31.472907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:31.471057    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:32.306959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:32.302143    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"
Oct 31 13:37:32.374174 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:32.374138    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"
Oct 31 13:37:32.625362 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:32.625266    4428 scope.go:122] "RemoveContainer" containerID="304f599b37ed35079b998f1616714763a8f0801ff8119e8c5ea44f5ba36bd713"
Oct 31 13:37:32.636650 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:32.636618    4428 scope.go:122] "RemoveContainer" containerID="4b40ef8be59b885389d176d71a073a3820c936c78f2523d505e6c5335e480854"
Oct 31 13:37:32.647387 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:32.647368    4428 scope.go:122] "RemoveContainer" containerID="f6b6296850d7f9a92093a7f85f5422992e8f94455a15aa04d7d75db10aa1d422"
Oct 31 13:37:32.656693 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:32.656673    4428 scope.go:122] "RemoveContainer" containerID="fb9c0312662263e78e83ea8625a142b46e9f37719c7c1675f4f47ad85d907151"
Oct 31 13:37:32.814899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:32.804882    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:33.920898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:33.918355    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:34.824898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:34.823909    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:35.694891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:35.694627    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:36.074046 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:36.074015    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:36.501095 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:36.501063    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:36.659523 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:36.659494    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:36.837977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:36.837893    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"]
Oct 31 13:37:36.845689 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:36.845660    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:36.857148 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:36.856558    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"]
Oct 31 13:37:36.912469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:36.912439    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d4b52a-openshift-default-dockercfg-2mlwj\""
Oct 31 13:37:36.944948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:36.944918    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:37.013892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.008358    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-workload-certs\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.013892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.008467    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h86x\" (UniqueName: \"kubernetes.io/projected/2ccec53f-f517-420f-b20d-2604a73f2884-kube-api-access-9h86x\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.013892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.008615    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ccec53f-f517-420f-b20d-2604a73f2884-istio-token\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.013892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.008671    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-credential-socket\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.013892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.008797    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ccec53f-f517-420f-b20d-2604a73f2884-istio-podinfo\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.013892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.008830    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-istio-envoy\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.013892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.008945    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-istio-data\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.013892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.008975    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-workload-socket\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.013892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.009055    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ccec53f-f517-420f-b20d-2604a73f2884-istiod-ca-cert\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.139073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.116413    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:37.139073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.117538    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ccec53f-f517-420f-b20d-2604a73f2884-istio-token\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.139073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.117597    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-credential-socket\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.139073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.117704    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ccec53f-f517-420f-b20d-2604a73f2884-istio-podinfo\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.139073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.117737    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-istio-envoy\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.139073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.117810    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-istio-data\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.139073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.117840    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-workload-socket\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.139073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.117927    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ccec53f-f517-420f-b20d-2604a73f2884-istiod-ca-cert\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.139073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.118024    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-workload-certs\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.139073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.118078    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h86x\" (UniqueName: \"kubernetes.io/projected/2ccec53f-f517-420f-b20d-2604a73f2884-kube-api-access-9h86x\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.139073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.126230    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-credential-socket\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.139073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.127007    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-workload-socket\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.175676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.151971    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-istio-data\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.175676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.152235    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-workload-certs\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.182032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.176596    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ccec53f-f517-420f-b20d-2604a73f2884-istio-token\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.195890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.189106    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ccec53f-f517-420f-b20d-2604a73f2884-istio-podinfo\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.195890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.190151    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h86x\" (UniqueName: \"kubernetes.io/projected/2ccec53f-f517-420f-b20d-2604a73f2884-kube-api-access-9h86x\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.195890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.194142    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ccec53f-f517-420f-b20d-2604a73f2884-istiod-ca-cert\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.195890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.195100    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-istio-envoy\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.478954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.478651    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:37.479836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.479784    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:37.514988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.511600    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"
Oct 31 13:37:37.648247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:37.648115    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:38.023960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.014745    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"]
Oct 31 13:37:38.130226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.127483    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:38.146931 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.136619    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:38.146931 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.136727    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:38.146931 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.136784    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:38.368191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.368114    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"]
Oct 31 13:37:38.373078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.373057    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"
Oct 31 13:37:38.379659 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.379635    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:37:38.379817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.379747    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:37:38.385571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.385550    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"]
Oct 31 13:37:38.417597 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.417565    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756a7df3-571b-4684-a369-01217173c597-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67\" (UID: \"756a7df3-571b-4684-a369-01217173c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"
Oct 31 13:37:38.417763 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.417700    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbbbx\" (UniqueName: \"kubernetes.io/projected/756a7df3-571b-4684-a369-01217173c597-kube-api-access-kbbbx\") pod \"maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67\" (UID: \"756a7df3-571b-4684-a369-01217173c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"
Oct 31 13:37:38.417763 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.417748    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756a7df3-571b-4684-a369-01217173c597-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67\" (UID: \"756a7df3-571b-4684-a369-01217173c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"
Oct 31 13:37:38.519211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.519170    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756a7df3-571b-4684-a369-01217173c597-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67\" (UID: \"756a7df3-571b-4684-a369-01217173c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"
Oct 31 13:37:38.519415 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.519258    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbbx\" (UniqueName: \"kubernetes.io/projected/756a7df3-571b-4684-a369-01217173c597-kube-api-access-kbbbx\") pod \"maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67\" (UID: \"756a7df3-571b-4684-a369-01217173c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"
Oct 31 13:37:38.519415 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.519295    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756a7df3-571b-4684-a369-01217173c597-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67\" (UID: \"756a7df3-571b-4684-a369-01217173c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"
Oct 31 13:37:38.519971 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.519946    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756a7df3-571b-4684-a369-01217173c597-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67\" (UID: \"756a7df3-571b-4684-a369-01217173c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"
Oct 31 13:37:38.521835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.521808    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756a7df3-571b-4684-a369-01217173c597-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67\" (UID: \"756a7df3-571b-4684-a369-01217173c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"
Oct 31 13:37:38.532708 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.532677    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbbbx\" (UniqueName: \"kubernetes.io/projected/756a7df3-571b-4684-a369-01217173c597-kube-api-access-kbbbx\") pod \"maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67\" (UID: \"756a7df3-571b-4684-a369-01217173c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"
Oct 31 13:37:38.694773 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.694697    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"
Oct 31 13:37:38.774476 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.774452    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"
Oct 31 13:37:38.983475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.983449    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"]
Oct 31 13:37:38.983676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.983642    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf" event={"ID":"2ccec53f-f517-420f-b20d-2604a73f2884","Type":"ContainerStarted","Data":"fddf8f8bc7bb0e249ae0e4d6dbd84845abd85605e4ca349ffb8d374bba279f5a"}
Oct 31 13:37:38.983736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:38.983689    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf" event={"ID":"2ccec53f-f517-420f-b20d-2604a73f2884","Type":"ContainerStarted","Data":"11262cf109eeed8cd66e3498cecc13ffc0241616e5a3a5687b5de954e9776c12"}
Oct 31 13:37:38.986456 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:37:38.986429    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756a7df3_571b_4684_a369_01217173c597.slice/crio-0fa7e84ca81e91c2a1e7a0b0c1ad5712b5b2a8de9de8a7eb07480e8175e75f51 WatchSource:0}: Error finding container 0fa7e84ca81e91c2a1e7a0b0c1ad5712b5b2a8de9de8a7eb07480e8175e75f51: Status 404 returned error can't find the container with id 0fa7e84ca81e91c2a1e7a0b0c1ad5712b5b2a8de9de8a7eb07480e8175e75f51
Oct 31 13:37:39.020429 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:39.020382    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf" podStartSLOduration=3.020368395 podStartE2EDuration="3.020368395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:39.0162202 +0000 UTC m=+2056.414568824" watchObservedRunningTime="2026-10-31 13:37:39.020368395 +0000 UTC m=+2056.418717019"
Oct 31 13:37:39.105098 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:39.105069    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:39.481360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:39.481334    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:39.487610 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:39.487583    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:39.847990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:39.847954    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:39.996292 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:39.996256    4428 generic.go:356] "Generic (PLEG): container finished" podID="756a7df3-571b-4684-a369-01217173c597" containerID="ec8d1e4ced2a227d8923e0abd4823f7c9bf59714a79d3fc849e884670c75edea" exitCode=0
Oct 31 13:37:39.996643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:39.996610    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67" event={"ID":"756a7df3-571b-4684-a369-01217173c597","Type":"ContainerDied","Data":"ec8d1e4ced2a227d8923e0abd4823f7c9bf59714a79d3fc849e884670c75edea"}
Oct 31 13:37:40.004745 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:39.996675    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67" event={"ID":"756a7df3-571b-4684-a369-01217173c597","Type":"ContainerStarted","Data":"0fa7e84ca81e91c2a1e7a0b0c1ad5712b5b2a8de9de8a7eb07480e8175e75f51"}
Oct 31 13:37:40.004745 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:39.996703    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:40.004745 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:39.998126    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:37:40.579230 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:40.579192    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:40.675569 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:40.675536    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"]
Oct 31 13:37:40.685758 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:40.685702    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6aa924-6lx56"]
Oct 31 13:37:40.748951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:40.748915    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:40.758191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:40.758166    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c24e33-d236-4bb6-adcb-3c2160c30b8d" path="/var/lib/kubelet/pods/60c24e33-d236-4bb6-adcb-3c2160c30b8d/volumes"
Oct 31 13:37:40.795499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:40.795478    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"
Oct 31 13:37:40.795902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:40.795858    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"
Oct 31 13:37:40.970596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:40.969509    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:41.183266 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.183234    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"
Oct 31 13:37:41.195338 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.195061    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"]
Oct 31 13:37:41.196629 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.196600    4428 cpu_manager.go:414] "containerMap: removing container" podUID="756a7df3-571b-4684-a369-01217173c597" containerName="revoke-keys"
Oct 31 13:37:41.196629 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.196630    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756a7df3-571b-4684-a369-01217173c597" containerName="revoke-keys"
Oct 31 13:37:41.196835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.196824    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="756a7df3-571b-4684-a369-01217173c597" containerName="revoke-keys"
Oct 31 13:37:41.201985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.201961    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.207156 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.207132    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-61da9d-openshift-default-dockercfg-nzt6j\""
Oct 31 13:37:41.215017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.214963    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"]
Oct 31 13:37:41.235722 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.235647    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.235950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.235923    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.236045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.236015    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.236107 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.236094    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.236199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.236182    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.236336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.236316    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.236408 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.236392    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.236516 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.236496    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.253128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.253094    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lkqp\" (UniqueName: \"kubernetes.io/projected/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-kube-api-access-4lkqp\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.260774 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.260747    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:41.354357 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.354320    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/756a7df3-571b-4684-a369-01217173c597-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756a7df3-571b-4684-a369-01217173c597-maas-api-xks-ca\") pod \"756a7df3-571b-4684-a369-01217173c597\" (UID: \"756a7df3-571b-4684-a369-01217173c597\") "
Oct 31 13:37:41.354633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.354602    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/756a7df3-571b-4684-a369-01217173c597-kube-api-access-kbbbx\" (UniqueName: \"kubernetes.io/projected/756a7df3-571b-4684-a369-01217173c597-kube-api-access-kbbbx\") pod \"756a7df3-571b-4684-a369-01217173c597\" (UID: \"756a7df3-571b-4684-a369-01217173c597\") "
Oct 31 13:37:41.354732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.354651    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/756a7df3-571b-4684-a369-01217173c597-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756a7df3-571b-4684-a369-01217173c597-maas-api-openshift-ca\") pod \"756a7df3-571b-4684-a369-01217173c597\" (UID: \"756a7df3-571b-4684-a369-01217173c597\") "
Oct 31 13:37:41.354918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.354900    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.355033 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.355015    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.355118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.355060    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.355118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.355083    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756a7df3-571b-4684-a369-01217173c597-maas-api-openshift-ca" pod "756a7df3-571b-4684-a369-01217173c597" (UID: "756a7df3-571b-4684-a369-01217173c597"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:41.356216 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.355437    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.356216 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.355528    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lkqp\" (UniqueName: \"kubernetes.io/projected/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-kube-api-access-4lkqp\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.356216 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.355608    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.356216 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.355624    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.356216 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.356011    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.356216 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.356095    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.356216 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.356152    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.356216 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.356169    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.358504 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.356569    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.358504 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.356613    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756a7df3-571b-4684-a369-01217173c597-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:41.360594 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.359093    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.392895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.374709    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.413159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.408305    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.413159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.408774    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756a7df3-571b-4684-a369-01217173c597-kube-api-access-kbbbx" pod "756a7df3-571b-4684-a369-01217173c597" (UID: "756a7df3-571b-4684-a369-01217173c597"). InnerVolumeSpecName "kube-api-access-kbbbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:41.413159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.410000    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/756a7df3-571b-4684-a369-01217173c597-maas-api-xks-ca" pod "756a7df3-571b-4684-a369-01217173c597" (UID: "756a7df3-571b-4684-a369-01217173c597"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:37:41.448035 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.445573    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lkqp\" (UniqueName: \"kubernetes.io/projected/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-kube-api-access-4lkqp\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.449845 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.449767    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:41.450021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.449899    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.479964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.458328    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbbbx\" (UniqueName: \"kubernetes.io/projected/756a7df3-571b-4684-a369-01217173c597-kube-api-access-kbbbx\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:41.479964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.458361    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756a7df3-571b-4684-a369-01217173c597-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:41.519582 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.519453    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:41.562201 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:41.561927    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:42.019685 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.019654    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"
Oct 31 13:37:42.019685 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.019737    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67" event={"ID":"756a7df3-571b-4684-a369-01217173c597","Type":"ContainerDied","Data":"0fa7e84ca81e91c2a1e7a0b0c1ad5712b5b2a8de9de8a7eb07480e8175e75f51"}
Oct 31 13:37:42.019685 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.019790    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fa7e84ca81e91c2a1e7a0b0c1ad5712b5b2a8de9de8a7eb07480e8175e75f51"
Oct 31 13:37:42.049122 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.049093    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:42.054567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.054526    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"]
Oct 31 13:37:42.055502 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:37:42.055413    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5efe5a3_93a2_4e34_ba6d_1037ac9b0954.slice/crio-f831ff3186b5c94d85d2445d7e17bec7f0785ca74cdeb3f0d17cf27df301a625 WatchSource:0}: Error finding container f831ff3186b5c94d85d2445d7e17bec7f0785ca74cdeb3f0d17cf27df301a625: Status 404 returned error can't find the container with id f831ff3186b5c94d85d2445d7e17bec7f0785ca74cdeb3f0d17cf27df301a625
Oct 31 13:37:42.058482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.058447    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:42.058589 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.058529    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:42.058589 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.058573    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:37:42.208794 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.208760    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"]
Oct 31 13:37:42.212589 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.212562    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:42.213960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.213939    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:37:42.219017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.218994    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f4232c-serving-cert\""
Oct 31 13:37:42.242544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.242499    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"]
Oct 31 13:37:42.286515 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.286477    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxr5f\" (UniqueName: \"kubernetes.io/projected/6738ac01-586f-4313-9ced-ccb4fbc5cd39-kube-api-access-cxr5f\") pod \"maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62\" (UID: \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:37:42.286712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.286583    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6738ac01-586f-4313-9ced-ccb4fbc5cd39-maas-api-tls\") pod \"maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62\" (UID: \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:37:42.286840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.286819    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6738ac01-586f-4313-9ced-ccb4fbc5cd39-metrics-certs\") pod \"maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62\" (UID: \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:37:42.286978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.286959    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6738ac01-586f-4313-9ced-ccb4fbc5cd39-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62\" (UID: \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:37:42.411291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.411187    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6738ac01-586f-4313-9ced-ccb4fbc5cd39-metrics-certs\") pod \"maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62\" (UID: \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:37:42.411692 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.411666    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6738ac01-586f-4313-9ced-ccb4fbc5cd39-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62\" (UID: \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:37:42.411915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.411897    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxr5f\" (UniqueName: \"kubernetes.io/projected/6738ac01-586f-4313-9ced-ccb4fbc5cd39-kube-api-access-cxr5f\") pod \"maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62\" (UID: \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:37:42.412207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.412170    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6738ac01-586f-4313-9ced-ccb4fbc5cd39-maas-api-tls\") pod \"maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62\" (UID: \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:37:42.417615 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.417586    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6738ac01-586f-4313-9ced-ccb4fbc5cd39-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62\" (UID: \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:37:42.437767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.437716    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6738ac01-586f-4313-9ced-ccb4fbc5cd39-metrics-certs\") pod \"maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62\" (UID: \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:37:42.493888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.482419    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6738ac01-586f-4313-9ced-ccb4fbc5cd39-maas-api-tls\") pod \"maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62\" (UID: \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:37:42.494920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.494758    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:42.497474 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.497414    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxr5f\" (UniqueName: \"kubernetes.io/projected/6738ac01-586f-4313-9ced-ccb4fbc5cd39-kube-api-access-cxr5f\") pod \"maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62\" (UID: \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:37:42.540704 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.538449    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:37:42.656252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.656190    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:42.847691 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:42.847661    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"]
Oct 31 13:37:42.848050 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:37:42.848019    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6738ac01_586f_4313_9ced_ccb4fbc5cd39.slice/crio-1266e6500e2033531249abd4eb7262c52d1817681dda7db4ec4cf7fa50b23e39 WatchSource:0}: Error finding container 1266e6500e2033531249abd4eb7262c52d1817681dda7db4ec4cf7fa50b23e39: Status 404 returned error can't find the container with id 1266e6500e2033531249abd4eb7262c52d1817681dda7db4ec4cf7fa50b23e39
Oct 31 13:37:43.078442 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.078389    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62" event={"ID":"6738ac01-586f-4313-9ced-ccb4fbc5cd39","Type":"ContainerStarted","Data":"1266e6500e2033531249abd4eb7262c52d1817681dda7db4ec4cf7fa50b23e39"}
Oct 31 13:37:43.110013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.108590    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww" event={"ID":"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954","Type":"ContainerStarted","Data":"65a8f3089a0bf9247fc8fa15579daf487383d3c6d94707994105b32bc9df7881"}
Oct 31 13:37:43.110013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.108684    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww" event={"ID":"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954","Type":"ContainerStarted","Data":"f831ff3186b5c94d85d2445d7e17bec7f0785ca74cdeb3f0d17cf27df301a625"}
Oct 31 13:37:43.155989 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.155939    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww" podStartSLOduration=2.1559243 podStartE2EDuration="2.1559243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:43.149013264 +0000 UTC m=+2060.547361901" watchObservedRunningTime="2026-10-31 13:37:43.1559243 +0000 UTC m=+2060.554273000"
Oct 31 13:37:43.557397 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.557350    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:43.562950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.562924    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:43.570257 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.570214    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:43.594987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.594953    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"]
Oct 31 13:37:43.602360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.602329    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"
Oct 31 13:37:43.607630 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.607596    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d4b52a-dockercfg-6xsjh\""
Oct 31 13:37:43.609995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.609972    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d4b52a\""
Oct 31 13:37:43.653040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.650063    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"]
Oct 31 13:37:43.670852 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.670814    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd12e2cb-e569-446e-8cbf-d6cd4779c833-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s\" (UID: \"dd12e2cb-e569-446e-8cbf-d6cd4779c833\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"
Oct 31 13:37:43.675354 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.671185    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9nzx\" (UniqueName: \"kubernetes.io/projected/dd12e2cb-e569-446e-8cbf-d6cd4779c833-kube-api-access-r9nzx\") pod \"payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s\" (UID: \"dd12e2cb-e569-446e-8cbf-d6cd4779c833\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"
Oct 31 13:37:43.809893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.782538    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd12e2cb-e569-446e-8cbf-d6cd4779c833-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s\" (UID: \"dd12e2cb-e569-446e-8cbf-d6cd4779c833\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"
Oct 31 13:37:43.809893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.782674    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9nzx\" (UniqueName: \"kubernetes.io/projected/dd12e2cb-e569-446e-8cbf-d6cd4779c833-kube-api-access-r9nzx\") pod \"payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s\" (UID: \"dd12e2cb-e569-446e-8cbf-d6cd4779c833\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"
Oct 31 13:37:43.817903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.811401    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd12e2cb-e569-446e-8cbf-d6cd4779c833-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s\" (UID: \"dd12e2cb-e569-446e-8cbf-d6cd4779c833\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"
Oct 31 13:37:43.838607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.838569    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:43.997857 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:43.990776    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9nzx\" (UniqueName: \"kubernetes.io/projected/dd12e2cb-e569-446e-8cbf-d6cd4779c833-kube-api-access-r9nzx\") pod \"payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s\" (UID: \"dd12e2cb-e569-446e-8cbf-d6cd4779c833\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"
Oct 31 13:37:44.022858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.022766    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"
Oct 31 13:37:44.050809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.050768    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"]
Oct 31 13:37:44.089905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.088765    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"
Oct 31 13:37:44.089905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.089563    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"]
Oct 31 13:37:44.126196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.126156    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngcnv\" (UniqueName: \"kubernetes.io/projected/9cf05b3f-e7f7-4b17-b06c-c015ae338bad-kube-api-access-ngcnv\") pod \"payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p\" (UID: \"9cf05b3f-e7f7-4b17-b06c-c015ae338bad\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"
Oct 31 13:37:44.126553 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.126529    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf05b3f-e7f7-4b17-b06c-c015ae338bad-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p\" (UID: \"9cf05b3f-e7f7-4b17-b06c-c015ae338bad\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"
Oct 31 13:37:44.137507 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.137463    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62" event={"ID":"6738ac01-586f-4313-9ced-ccb4fbc5cd39","Type":"ContainerStarted","Data":"18865331237ab4c2257453b479895e3fcfca14299f08a55d8b749c9d12118472"}
Oct 31 13:37:44.138958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.138240    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:44.139404 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.139386    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:37:44.158896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.157781    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:37:44.207891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.186075    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:44.238047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.237979    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62" podStartSLOduration=1.7098546300000002 podStartE2EDuration="2.237954966s" totalImagesPullingTime="528.100336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:44.21342255 +0000 UTC m=+2061.611771177" watchObservedRunningTime="2026-10-31 13:37:44.237954966 +0000 UTC m=+2061.636303591"
Oct 31 13:37:44.290941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.287819    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngcnv\" (UniqueName: \"kubernetes.io/projected/9cf05b3f-e7f7-4b17-b06c-c015ae338bad-kube-api-access-ngcnv\") pod \"payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p\" (UID: \"9cf05b3f-e7f7-4b17-b06c-c015ae338bad\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"
Oct 31 13:37:44.290941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.288050    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf05b3f-e7f7-4b17-b06c-c015ae338bad-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p\" (UID: \"9cf05b3f-e7f7-4b17-b06c-c015ae338bad\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"
Oct 31 13:37:44.292394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.292366    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf05b3f-e7f7-4b17-b06c-c015ae338bad-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p\" (UID: \"9cf05b3f-e7f7-4b17-b06c-c015ae338bad\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"
Oct 31 13:37:44.309747 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.309702    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngcnv\" (UniqueName: \"kubernetes.io/projected/9cf05b3f-e7f7-4b17-b06c-c015ae338bad-kube-api-access-ngcnv\") pod \"payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p\" (UID: \"9cf05b3f-e7f7-4b17-b06c-c015ae338bad\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"
Oct 31 13:37:44.494032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.493947    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"
Oct 31 13:37:44.610186 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.610091    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"]
Oct 31 13:37:44.611474 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:37:44.611447    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd12e2cb_e569_446e_8cbf_d6cd4779c833.slice/crio-9d118625d7c48e18413fff1a918b002218ca122219ad33ddbeff6f8ac7601e79 WatchSource:0}: Error finding container 9d118625d7c48e18413fff1a918b002218ca122219ad33ddbeff6f8ac7601e79: Status 404 returned error can't find the container with id 9d118625d7c48e18413fff1a918b002218ca122219ad33ddbeff6f8ac7601e79
Oct 31 13:37:44.817517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:44.817486    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"]
Oct 31 13:37:44.823508 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:37:44.823484    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf05b3f_e7f7_4b17_b06c_c015ae338bad.slice/crio-a7192d195c0b589768977ad1c1d54efb8fb900b34f46ca47153420b5c5c22c6e WatchSource:0}: Error finding container a7192d195c0b589768977ad1c1d54efb8fb900b34f46ca47153420b5c5c22c6e: Status 404 returned error can't find the container with id a7192d195c0b589768977ad1c1d54efb8fb900b34f46ca47153420b5c5c22c6e
Oct 31 13:37:45.146450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:45.146361    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p" event={"ID":"9cf05b3f-e7f7-4b17-b06c-c015ae338bad","Type":"ContainerStarted","Data":"f4af763bfc4b28c4c6b9ad53de8620dee77c08acb22ba5930134797bed7dc5e6"}
Oct 31 13:37:45.146450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:45.146427    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p" event={"ID":"9cf05b3f-e7f7-4b17-b06c-c015ae338bad","Type":"ContainerStarted","Data":"a7192d195c0b589768977ad1c1d54efb8fb900b34f46ca47153420b5c5c22c6e"}
Oct 31 13:37:45.146989 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:45.146459    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"
Oct 31 13:37:45.148121 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:45.148098    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s" event={"ID":"dd12e2cb-e569-446e-8cbf-d6cd4779c833","Type":"ContainerStarted","Data":"765142255aff4fff167f6035389dd098a29fed8cb1425591e6107036ff29c1ee"}
Oct 31 13:37:45.148244 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:45.148141    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s" event={"ID":"dd12e2cb-e569-446e-8cbf-d6cd4779c833","Type":"ContainerStarted","Data":"9d118625d7c48e18413fff1a918b002218ca122219ad33ddbeff6f8ac7601e79"}
Oct 31 13:37:45.148449 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:45.148428    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"
Oct 31 13:37:45.168500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:45.168448    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p" podStartSLOduration=2.16843566 podStartE2EDuration="2.16843566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:45.166406502 +0000 UTC m=+2062.564755127" watchObservedRunningTime="2026-10-31 13:37:45.16843566 +0000 UTC m=+2062.566784284"
Oct 31 13:37:45.187069 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:45.187022    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s" podStartSLOduration=2.187004808 podStartE2EDuration="2.187004808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:45.185425185 +0000 UTC m=+2062.583773811" watchObservedRunningTime="2026-10-31 13:37:45.187004808 +0000 UTC m=+2062.585353433"
Oct 31 13:37:45.339467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:45.339423    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:45.668058 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:45.668013    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:45.829836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:45.829798    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:46.433893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:46.433562    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:46.840958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:46.840929    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"]
Oct 31 13:37:46.845668 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:46.845647    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"
Oct 31 13:37:46.853842 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:46.853817    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-61da9d\""
Oct 31 13:37:46.855212 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:46.855056    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:46.858914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:46.858856    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"]
Oct 31 13:37:46.887407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:46.887357    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8cf8de-bfa2-4e2d-8069-e1b1132fe426-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl\" (UID: \"5d8cf8de-bfa2-4e2d-8069-e1b1132fe426\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"
Oct 31 13:37:46.887598 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:46.887442    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b4md\" (UniqueName: \"kubernetes.io/projected/5d8cf8de-bfa2-4e2d-8069-e1b1132fe426-kube-api-access-9b4md\") pod \"payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl\" (UID: \"5d8cf8de-bfa2-4e2d-8069-e1b1132fe426\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"
Oct 31 13:37:46.924496 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:46.924460    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"]
Oct 31 13:37:46.929316 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:46.929294    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"
Oct 31 13:37:46.938694 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:46.938652    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"]
Oct 31 13:37:47.049897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.040298    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-61da9d-dockercfg-2vwvz\""
Oct 31 13:37:47.049897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.043617    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530cfbed-6986-4f00-9eb5-bb0d7985827e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t\" (UID: \"530cfbed-6986-4f00-9eb5-bb0d7985827e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"
Oct 31 13:37:47.107947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.095391    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bks5c\" (UniqueName: \"kubernetes.io/projected/530cfbed-6986-4f00-9eb5-bb0d7985827e-kube-api-access-bks5c\") pod \"payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t\" (UID: \"530cfbed-6986-4f00-9eb5-bb0d7985827e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"
Oct 31 13:37:47.107947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.095475    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8cf8de-bfa2-4e2d-8069-e1b1132fe426-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl\" (UID: \"5d8cf8de-bfa2-4e2d-8069-e1b1132fe426\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"
Oct 31 13:37:47.107947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.095546    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b4md\" (UniqueName: \"kubernetes.io/projected/5d8cf8de-bfa2-4e2d-8069-e1b1132fe426-kube-api-access-9b4md\") pod \"payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl\" (UID: \"5d8cf8de-bfa2-4e2d-8069-e1b1132fe426\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"
Oct 31 13:37:47.107947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.096846    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8cf8de-bfa2-4e2d-8069-e1b1132fe426-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl\" (UID: \"5d8cf8de-bfa2-4e2d-8069-e1b1132fe426\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"
Oct 31 13:37:47.207856 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.146492    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:47.230892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.225057    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530cfbed-6986-4f00-9eb5-bb0d7985827e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t\" (UID: \"530cfbed-6986-4f00-9eb5-bb0d7985827e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"
Oct 31 13:37:47.230892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.225345    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bks5c\" (UniqueName: \"kubernetes.io/projected/530cfbed-6986-4f00-9eb5-bb0d7985827e-kube-api-access-bks5c\") pod \"payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t\" (UID: \"530cfbed-6986-4f00-9eb5-bb0d7985827e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"
Oct 31 13:37:47.245805 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.245772    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530cfbed-6986-4f00-9eb5-bb0d7985827e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t\" (UID: \"530cfbed-6986-4f00-9eb5-bb0d7985827e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"
Oct 31 13:37:47.288268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.283530    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b4md\" (UniqueName: \"kubernetes.io/projected/5d8cf8de-bfa2-4e2d-8069-e1b1132fe426-kube-api-access-9b4md\") pod \"payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl\" (UID: \"5d8cf8de-bfa2-4e2d-8069-e1b1132fe426\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"
Oct 31 13:37:47.292830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.292732    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bks5c\" (UniqueName: \"kubernetes.io/projected/530cfbed-6986-4f00-9eb5-bb0d7985827e-kube-api-access-bks5c\") pod \"payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t\" (UID: \"530cfbed-6986-4f00-9eb5-bb0d7985827e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"
Oct 31 13:37:47.465169 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.465136    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"
Oct 31 13:37:47.550178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.550149    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"
Oct 31 13:37:47.664438 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.663422    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"]
Oct 31 13:37:47.670168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.670147    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:37:47.672896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.672849    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"]
Oct 31 13:37:47.675672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.675651    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fb50fa-serving-cert\""
Oct 31 13:37:47.789973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.779173    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:47.876894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.851132    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/511e4897-ed7c-463b-a199-1e48519db98a-maas-api-tls\") pod \"maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h\" (UID: \"511e4897-ed7c-463b-a199-1e48519db98a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:37:47.876894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.851191    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsx7c\" (UniqueName: \"kubernetes.io/projected/511e4897-ed7c-463b-a199-1e48519db98a-kube-api-access-rsx7c\") pod \"maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h\" (UID: \"511e4897-ed7c-463b-a199-1e48519db98a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:37:47.876894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.851335    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/511e4897-ed7c-463b-a199-1e48519db98a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h\" (UID: \"511e4897-ed7c-463b-a199-1e48519db98a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:37:47.876894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.852472    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"]
Oct 31 13:37:47.876894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.851452    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/511e4897-ed7c-463b-a199-1e48519db98a-metrics-certs\") pod \"maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h\" (UID: \"511e4897-ed7c-463b-a199-1e48519db98a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:37:47.933768 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:37:47.933730    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530cfbed_6986_4f00_9eb5_bb0d7985827e.slice/crio-a6b6853ca239c47493ca2904bc583258b1e95fcfb28d92af2ac2772037e4fe2c WatchSource:0}: Error finding container a6b6853ca239c47493ca2904bc583258b1e95fcfb28d92af2ac2772037e4fe2c: Status 404 returned error can't find the container with id a6b6853ca239c47493ca2904bc583258b1e95fcfb28d92af2ac2772037e4fe2c
Oct 31 13:37:47.936424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.936394    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"]
Oct 31 13:37:47.953507 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.953418    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:47.955251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.955226    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsx7c\" (UniqueName: \"kubernetes.io/projected/511e4897-ed7c-463b-a199-1e48519db98a-kube-api-access-rsx7c\") pod \"maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h\" (UID: \"511e4897-ed7c-463b-a199-1e48519db98a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:37:47.955392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.955373    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/511e4897-ed7c-463b-a199-1e48519db98a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h\" (UID: \"511e4897-ed7c-463b-a199-1e48519db98a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:37:47.955477 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.955463    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/511e4897-ed7c-463b-a199-1e48519db98a-metrics-certs\") pod \"maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h\" (UID: \"511e4897-ed7c-463b-a199-1e48519db98a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:37:47.955635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.955612    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/511e4897-ed7c-463b-a199-1e48519db98a-maas-api-tls\") pod \"maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h\" (UID: \"511e4897-ed7c-463b-a199-1e48519db98a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:37:47.956630 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.956597    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/511e4897-ed7c-463b-a199-1e48519db98a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h\" (UID: \"511e4897-ed7c-463b-a199-1e48519db98a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:37:47.958685 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.958569    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/511e4897-ed7c-463b-a199-1e48519db98a-maas-api-tls\") pod \"maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h\" (UID: \"511e4897-ed7c-463b-a199-1e48519db98a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:37:47.958917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.958858    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/511e4897-ed7c-463b-a199-1e48519db98a-metrics-certs\") pod \"maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h\" (UID: \"511e4897-ed7c-463b-a199-1e48519db98a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:37:47.968625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.968603    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsx7c\" (UniqueName: \"kubernetes.io/projected/511e4897-ed7c-463b-a199-1e48519db98a-kube-api-access-rsx7c\") pod \"maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h\" (UID: \"511e4897-ed7c-463b-a199-1e48519db98a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:37:47.988163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:47.988143    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:37:48.396399 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:48.392459    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl" event={"ID":"5d8cf8de-bfa2-4e2d-8069-e1b1132fe426","Type":"ContainerStarted","Data":"612cee8557ca690e42db9e275a8b012bea92750513f5fd2ab4bb6675d53403b5"}
Oct 31 13:37:48.396399 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:48.392546    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl" event={"ID":"5d8cf8de-bfa2-4e2d-8069-e1b1132fe426","Type":"ContainerStarted","Data":"ca14b25de3f8034fe7df2ea8f379609d382e6f05d99650a9b9a10af055f0bd56"}
Oct 31 13:37:48.396399 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:48.393342    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"
Oct 31 13:37:48.404646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:48.404599    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t" event={"ID":"530cfbed-6986-4f00-9eb5-bb0d7985827e","Type":"ContainerStarted","Data":"86bb0549b47fbe808d32b79a071312394d443405c1fa078cb82c5ade407dfc90"}
Oct 31 13:37:48.404816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:48.404660    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t" event={"ID":"530cfbed-6986-4f00-9eb5-bb0d7985827e","Type":"ContainerStarted","Data":"a6b6853ca239c47493ca2904bc583258b1e95fcfb28d92af2ac2772037e4fe2c"}
Oct 31 13:37:48.404990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:48.404972    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"
Oct 31 13:37:48.422855 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:48.422824    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"]
Oct 31 13:37:48.424334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:48.424290    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl" podStartSLOduration=2.424275034 podStartE2EDuration="2.424275034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:48.419750367 +0000 UTC m=+2065.818099005" watchObservedRunningTime="2026-10-31 13:37:48.424275034 +0000 UTC m=+2065.822623661"
Oct 31 13:37:48.430660 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:37:48.430637    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511e4897_ed7c_463b_a199_1e48519db98a.slice/crio-527f89ddaff3b1b7f6d5c9bfb9b2cab2c3d5dd6c1987f738a65a73de3cbabdff WatchSource:0}: Error finding container 527f89ddaff3b1b7f6d5c9bfb9b2cab2c3d5dd6c1987f738a65a73de3cbabdff: Status 404 returned error can't find the container with id 527f89ddaff3b1b7f6d5c9bfb9b2cab2c3d5dd6c1987f738a65a73de3cbabdff
Oct 31 13:37:48.446023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:48.445977    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t" podStartSLOduration=2.44596473 podStartE2EDuration="2.44596473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:48.442260157 +0000 UTC m=+2065.840608781" watchObservedRunningTime="2026-10-31 13:37:48.44596473 +0000 UTC m=+2065.844313357"
Oct 31 13:37:48.625904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:48.624950    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"]
Oct 31 13:37:48.721855 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:48.721820    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:48.752728 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:37:48.752695    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:37:48.883220 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:48.883190    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:49.095943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:49.090759    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:49.416202 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:49.416108    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h" event={"ID":"511e4897-ed7c-463b-a199-1e48519db98a","Type":"ContainerStarted","Data":"3a5ddf61d140a274fb85057f0ff398ad997958bd8982717a9339a3784fa6212f"}
Oct 31 13:37:49.416599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:49.416577    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:37:49.416667 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:49.416631    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h" event={"ID":"511e4897-ed7c-463b-a199-1e48519db98a","Type":"ContainerStarted","Data":"527f89ddaff3b1b7f6d5c9bfb9b2cab2c3d5dd6c1987f738a65a73de3cbabdff"}
Oct 31 13:37:49.448820 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:49.448757    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h" podStartSLOduration=2.020404092 podStartE2EDuration="2.448738626s" totalImagesPullingTime="428.334534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:49.442568169 +0000 UTC m=+2066.840916796" watchObservedRunningTime="2026-10-31 13:37:49.448738626 +0000 UTC m=+2066.847087243"
Oct 31 13:37:50.063079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:50.048473    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:50.217736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:50.213759    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:50.372139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:50.372011    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:51.251930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:51.246216    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:37:51.294430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:51.294394    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:51.532014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:51.527791    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:51.817328 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:51.817234    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:53.073833 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:53.073782    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:53.625423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:53.625384    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k" podUID="093cde47-6a66-4a54-8ee4-6c205794b2f4" containerName="payload-processing" containerID="cri-o://db01943d22177f5c12f5bffb8fe8344171ac2d74639aed03fbc60dffc93e2db0" gracePeriod=25
Oct 31 13:37:54.093125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.092741    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"
Oct 31 13:37:54.121764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.121724    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/093cde47-6a66-4a54-8ee4-6c205794b2f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/093cde47-6a66-4a54-8ee4-6c205794b2f4-plugins-config-volume\") pod \"093cde47-6a66-4a54-8ee4-6c205794b2f4\" (UID: \"093cde47-6a66-4a54-8ee4-6c205794b2f4\") "
Oct 31 13:37:54.121764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.121769    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093cde47-6a66-4a54-8ee4-6c205794b2f4-kube-api-access-6xcnt\" (UniqueName: \"kubernetes.io/projected/093cde47-6a66-4a54-8ee4-6c205794b2f4-kube-api-access-6xcnt\") pod \"093cde47-6a66-4a54-8ee4-6c205794b2f4\" (UID: \"093cde47-6a66-4a54-8ee4-6c205794b2f4\") "
Oct 31 13:37:54.123092 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.123058    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093cde47-6a66-4a54-8ee4-6c205794b2f4-plugins-config-volume" pod "093cde47-6a66-4a54-8ee4-6c205794b2f4" (UID: "093cde47-6a66-4a54-8ee4-6c205794b2f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:37:54.125037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.125010    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093cde47-6a66-4a54-8ee4-6c205794b2f4-kube-api-access-6xcnt" pod "093cde47-6a66-4a54-8ee4-6c205794b2f4" (UID: "093cde47-6a66-4a54-8ee4-6c205794b2f4"). InnerVolumeSpecName "kube-api-access-6xcnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:37:54.283949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.251938    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:54.301344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.301309    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/093cde47-6a66-4a54-8ee4-6c205794b2f4-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:54.301587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.301570    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xcnt\" (UniqueName: \"kubernetes.io/projected/093cde47-6a66-4a54-8ee4-6c205794b2f4-kube-api-access-6xcnt\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:37:54.616068 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.615975    4428 generic.go:356] "Generic (PLEG): container finished" podID="093cde47-6a66-4a54-8ee4-6c205794b2f4" containerID="db01943d22177f5c12f5bffb8fe8344171ac2d74639aed03fbc60dffc93e2db0" exitCode=0
Oct 31 13:37:54.616068 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.616020    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"
Oct 31 13:37:54.616318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.616022    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k" event={"ID":"093cde47-6a66-4a54-8ee4-6c205794b2f4","Type":"ContainerDied","Data":"db01943d22177f5c12f5bffb8fe8344171ac2d74639aed03fbc60dffc93e2db0"}
Oct 31 13:37:54.616318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.616142    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k" event={"ID":"093cde47-6a66-4a54-8ee4-6c205794b2f4","Type":"ContainerDied","Data":"5ce1bcf512f62f140df260e3d3f7110003e448e619b506049e6c095229e01a8c"}
Oct 31 13:37:54.616318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.616172    4428 scope.go:122] "RemoveContainer" containerID="db01943d22177f5c12f5bffb8fe8344171ac2d74639aed03fbc60dffc93e2db0"
Oct 31 13:37:54.631055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.630828    4428 scope.go:122] "RemoveContainer" containerID="db01943d22177f5c12f5bffb8fe8344171ac2d74639aed03fbc60dffc93e2db0"
Oct 31 13:37:54.631192 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:37:54.631171    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db01943d22177f5c12f5bffb8fe8344171ac2d74639aed03fbc60dffc93e2db0\": container with ID starting with db01943d22177f5c12f5bffb8fe8344171ac2d74639aed03fbc60dffc93e2db0 not found: ID does not exist" containerID="db01943d22177f5c12f5bffb8fe8344171ac2d74639aed03fbc60dffc93e2db0"
Oct 31 13:37:54.631267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.631204    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db01943d22177f5c12f5bffb8fe8344171ac2d74639aed03fbc60dffc93e2db0"} err="failed to get container status \"db01943d22177f5c12f5bffb8fe8344171ac2d74639aed03fbc60dffc93e2db0\": rpc error: code = NotFound desc = could not find container \"db01943d22177f5c12f5bffb8fe8344171ac2d74639aed03fbc60dffc93e2db0\": container with ID starting with db01943d22177f5c12f5bffb8fe8344171ac2d74639aed03fbc60dffc93e2db0 not found: ID does not exist"
Oct 31 13:37:54.670895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.650285    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"]
Oct 31 13:37:54.698781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.698733    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbb3d5-dbccc5d5f-z2p7k"]
Oct 31 13:37:54.806356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:54.806301    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093cde47-6a66-4a54-8ee4-6c205794b2f4" path="/var/lib/kubelet/pods/093cde47-6a66-4a54-8ee4-6c205794b2f4/volumes"
Oct 31 13:37:55.160891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:55.155563    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:55.437085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:55.437001    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:37:56.224267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:56.224231    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"
Oct 31 13:37:56.241909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:56.241592    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:56.268895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:56.262577    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"
Oct 31 13:37:57.192503 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:57.192465    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:58.145312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:58.145252    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:58.845914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:58.845719    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:37:59.420436 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:59.420405    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"
Oct 31 13:37:59.421015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:59.420531    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"
Oct 31 13:37:59.650799 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:37:59.650749    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:00.409742 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:00.409639    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:01.237590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:01.237465    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:02.126678 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:02.126580    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:02.214553 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:02.214498    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"]
Oct 31 13:38:02.369822 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:02.369781    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:02.572903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:02.562423    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:03.153608 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:03.153565    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:03.321293 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:03.321257    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:03.872811 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:03.872713    4428 generic.go:356] "Generic (PLEG): container finished" podID="c080a436-0943-4c33-938c-35b1ff2c1a11" containerID="d6ebff4bee6e489501f3c38883f9f91780938f8f20ade5c84b5c7d5ac26085a8" exitCode=0
Oct 31 13:38:03.873775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:03.872979    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x" event={"ID":"c080a436-0943-4c33-938c-35b1ff2c1a11","Type":"ContainerDied","Data":"d6ebff4bee6e489501f3c38883f9f91780938f8f20ade5c84b5c7d5ac26085a8"}
Oct 31 13:38:03.873775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:03.873623    4428 scope.go:122] "RemoveContainer" containerID="d6ebff4bee6e489501f3c38883f9f91780938f8f20ade5c84b5c7d5ac26085a8"
Oct 31 13:38:03.876939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:03.876908    4428 generic.go:356] "Generic (PLEG): container finished" podID="b290805f-2d02-4d45-abbd-d0f0f8afe535" containerID="2e621b67784052dd20cecfbcbb2092d3c74843eae22e2a62a4f98e6b7f109989" exitCode=0
Oct 31 13:38:03.877108 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:03.877041    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:03.877752 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:03.877210    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:03.877752 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:03.877242    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb" event={"ID":"b290805f-2d02-4d45-abbd-d0f0f8afe535","Type":"ContainerDied","Data":"2e621b67784052dd20cecfbcbb2092d3c74843eae22e2a62a4f98e6b7f109989"}
Oct 31 13:38:03.877752 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:03.877324    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:03.878084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:03.878051    4428 scope.go:122] "RemoveContainer" containerID="2e621b67784052dd20cecfbcbb2092d3c74843eae22e2a62a4f98e6b7f109989"
Oct 31 13:38:03.882211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:03.882181    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:03.882309 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:03.882253    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:03.882309 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:03.882287    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:04.215946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:04.215080    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:04.888521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:04.888483    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x" event={"ID":"c080a436-0943-4c33-938c-35b1ff2c1a11","Type":"ContainerStarted","Data":"c6699006304b01ee77cdd77437b3100340400b010222ebb90ddbb9343f322c9b"}
Oct 31 13:38:04.890915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:04.890886    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb" event={"ID":"b290805f-2d02-4d45-abbd-d0f0f8afe535","Type":"ContainerStarted","Data":"9b68c96b43dc262c1dd85a9d0c4032fbc64d8b083dfd3033a56d13039f9dba34"}
Oct 31 13:38:05.097899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:05.094046    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:38:05.097899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:05.094209    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 13:38:05.097899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:05.094298    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb" podUID="b290805f-2d02-4d45-abbd-d0f0f8afe535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 13:38:05.332278 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:05.332247    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:38:05.333620 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:05.333589    4428 patch_prober.go:29] interesting pod/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 31 13:38:05.333765 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:05.333644    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x" podUID="c080a436-0943-4c33-938c-35b1ff2c1a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 31 13:38:05.436824 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:05.436791    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:06.093476 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:06.093439    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 13:38:06.094472 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:06.094010    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb" podUID="b290805f-2d02-4d45-abbd-d0f0f8afe535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 13:38:06.319213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:06.311797    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:06.554959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:06.398159    4428 patch_prober.go:29] interesting pod/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 31 13:38:06.554959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:06.398225    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x" podUID="c080a436-0943-4c33-938c-35b1ff2c1a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 31 13:38:06.580897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:06.579535    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:06.800779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:06.800746    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:07.101890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:07.099598    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 13:38:07.101890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:07.099669    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb" podUID="b290805f-2d02-4d45-abbd-d0f0f8afe535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 13:38:07.218897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:07.218209    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj" podUID="8a7e177f-205f-4ba6-8711-40a0b52158ea" containerName="payload-pre-processing" containerID="cri-o://4902bf17e7985d36132b6f596a5436dc6901b48efd6a339eeeffba2e73941c29" gracePeriod=25
Oct 31 13:38:07.311889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:07.311831    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:07.333334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:07.333291    4428 patch_prober.go:29] interesting pod/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 31 13:38:07.333610 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:07.333578    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x" podUID="c080a436-0943-4c33-938c-35b1ff2c1a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 31 13:38:07.489897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:07.481009    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:07.913560 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:07.913522    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:08.028467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:08.028433    4428 generic.go:356] "Generic (PLEG): container finished" podID="8a7e177f-205f-4ba6-8711-40a0b52158ea" containerID="4902bf17e7985d36132b6f596a5436dc6901b48efd6a339eeeffba2e73941c29" exitCode=0
Oct 31 13:38:08.028733 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:08.028716    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj" event={"ID":"8a7e177f-205f-4ba6-8711-40a0b52158ea","Type":"ContainerDied","Data":"4902bf17e7985d36132b6f596a5436dc6901b48efd6a339eeeffba2e73941c29"}
Oct 31 13:38:08.101023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:08.100485    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 13:38:08.101023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:08.100552    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb" podUID="b290805f-2d02-4d45-abbd-d0f0f8afe535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 13:38:08.138919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:08.138896    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"
Oct 31 13:38:08.174970 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:08.172225    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:08.230933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:08.217183    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a7e177f-205f-4ba6-8711-40a0b52158ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7e177f-205f-4ba6-8711-40a0b52158ea-plugins-config-volume\") pod \"8a7e177f-205f-4ba6-8711-40a0b52158ea\" (UID: \"8a7e177f-205f-4ba6-8711-40a0b52158ea\") "
Oct 31 13:38:08.230933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:08.217535    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7e177f-205f-4ba6-8711-40a0b52158ea-kube-api-access-mpjpq\" (UniqueName: \"kubernetes.io/projected/8a7e177f-205f-4ba6-8711-40a0b52158ea-kube-api-access-mpjpq\") pod \"8a7e177f-205f-4ba6-8711-40a0b52158ea\" (UID: \"8a7e177f-205f-4ba6-8711-40a0b52158ea\") "
Oct 31 13:38:08.237904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:08.231773    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7e177f-205f-4ba6-8711-40a0b52158ea-plugins-config-volume" pod "8a7e177f-205f-4ba6-8711-40a0b52158ea" (UID: "8a7e177f-205f-4ba6-8711-40a0b52158ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:38:08.322276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:08.322239    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7e177f-205f-4ba6-8711-40a0b52158ea-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:38:08.387893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:08.387191    4428 patch_prober.go:29] interesting pod/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 31 13:38:08.387893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:08.387259    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x" podUID="c080a436-0943-4c33-938c-35b1ff2c1a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 31 13:38:08.412891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:08.388503    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7e177f-205f-4ba6-8711-40a0b52158ea-kube-api-access-mpjpq" pod "8a7e177f-205f-4ba6-8711-40a0b52158ea" (UID: "8a7e177f-205f-4ba6-8711-40a0b52158ea"). InnerVolumeSpecName "kube-api-access-mpjpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:38:08.448142 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:08.448050    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpjpq\" (UniqueName: \"kubernetes.io/projected/8a7e177f-205f-4ba6-8711-40a0b52158ea-kube-api-access-mpjpq\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:38:08.638180 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:08.628198    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:38:08.638180 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:08.629766    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:08.918907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:08.918635    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:09.305904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.300747    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"
Oct 31 13:38:09.313915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.308328    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:38:09.315757 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.315708    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:38:09.343223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.343160    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:09.349363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.349321    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 13:38:09.350400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.350367    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb" podUID="b290805f-2d02-4d45-abbd-d0f0f8afe535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 13:38:09.374939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.352343    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"]
Oct 31 13:38:09.383240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.383203    4428 cpu_manager.go:414] "containerMap: removing container" podUID="8a7e177f-205f-4ba6-8711-40a0b52158ea" containerName="payload-pre-processing"
Oct 31 13:38:09.383470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.383453    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7e177f-205f-4ba6-8711-40a0b52158ea" containerName="payload-pre-processing"
Oct 31 13:38:09.383603 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.383589    4428 cpu_manager.go:414] "containerMap: removing container" podUID="093cde47-6a66-4a54-8ee4-6c205794b2f4" containerName="payload-processing"
Oct 31 13:38:09.383697 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.383684    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093cde47-6a66-4a54-8ee4-6c205794b2f4" containerName="payload-processing"
Oct 31 13:38:09.389897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.384117    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7e177f-205f-4ba6-8711-40a0b52158ea" containerName="payload-pre-processing"
Oct 31 13:38:09.390163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.390141    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="093cde47-6a66-4a54-8ee4-6c205794b2f4" containerName="payload-processing"
Oct 31 13:38:09.398015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.397985    4428 patch_prober.go:29] interesting pod/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 31 13:38:09.398617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.398583    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x" podUID="c080a436-0943-4c33-938c-35b1ff2c1a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 31 13:38:09.476896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.461034    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:38:09.476896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.462758    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:38:09.479132 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.477723    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:09.527826 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.527791    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"]
Oct 31 13:38:09.528625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.528581    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj" event={"ID":"8a7e177f-205f-4ba6-8711-40a0b52158ea","Type":"ContainerDied","Data":"5b646827e64bf81ea0d960bda71727f80567c32be8c9e854870770b00109585e"}
Oct 31 13:38:09.528816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.528801    4428 scope.go:122] "RemoveContainer" containerID="4902bf17e7985d36132b6f596a5436dc6901b48efd6a339eeeffba2e73941c29"
Oct 31 13:38:09.533723 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.528163    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.557663 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.557577    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cae9ec-openshift-default-dockercfg-pdrhx\""
Oct 31 13:38:09.600959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.600921    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"]
Oct 31 13:38:09.623578 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.623550    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c45a37-7b55b8cdf9-tzswj"]
Oct 31 13:38:09.697195 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.697145    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istiod-ca-cert\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.697421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.697367    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-podinfo\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.697501 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.697458    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-credential-socket\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.697579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.697557    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-data\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.697634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.697604    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sbsx\" (UniqueName: \"kubernetes.io/projected/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-kube-api-access-9sbsx\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.697685 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.697654    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-token\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.697736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.697709    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-workload-certs\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.697796 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.697761    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-workload-socket\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.697796 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.697784    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-envoy\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.799958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.799915    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-podinfo\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.801093 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.801054    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-credential-socket\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.801235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.801177    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-data\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.801235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.801216    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sbsx\" (UniqueName: \"kubernetes.io/projected/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-kube-api-access-9sbsx\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.801379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.801253    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-token\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.801379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.801293    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-workload-certs\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.801379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.801326    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-workload-socket\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.801379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.801342    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-envoy\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.801659 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.801440    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istiod-ca-cert\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.802018 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.801987    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-credential-socket\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.802129 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.802051    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-data\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.802512 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.802353    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istiod-ca-cert\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.802512 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.802420    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-workload-socket\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.802698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.802587    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-workload-certs\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.803919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.803898    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-podinfo\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.805608 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.805585    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-envoy\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.815959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.815480    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sbsx\" (UniqueName: \"kubernetes.io/projected/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-kube-api-access-9sbsx\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.815959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.815706    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-token\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:09.940106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:09.940069    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:10.092952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:10.092526    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 13:38:10.092952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:10.092586    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb" podUID="b290805f-2d02-4d45-abbd-d0f0f8afe535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 13:38:10.256895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:10.250203    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:38:10.362955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:10.362207    4428 patch_prober.go:29] interesting pod/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 31 13:38:10.362955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:10.362272    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x" podUID="c080a436-0943-4c33-938c-35b1ff2c1a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 31 13:38:10.797921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:10.797708    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:10.841810 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:10.841773    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7e177f-205f-4ba6-8711-40a0b52158ea" path="/var/lib/kubelet/pods/8a7e177f-205f-4ba6-8711-40a0b52158ea/volumes"
Oct 31 13:38:11.006673 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:38:11.006633    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1af67f4_a0eb_402b_8453_7cb5ab463ab1.slice/crio-8852e6431856e545e349c6fd32ebf3c69f91057f7ed4b6d9cc67ab643c7321c6 WatchSource:0}: Error finding container 8852e6431856e545e349c6fd32ebf3c69f91057f7ed4b6d9cc67ab643c7321c6: Status 404 returned error can't find the container with id 8852e6431856e545e349c6fd32ebf3c69f91057f7ed4b6d9cc67ab643c7321c6
Oct 31 13:38:11.012497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:11.012451    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:11.012591 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:11.012552    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:11.012631 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:11.012598    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:11.014564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:11.014534    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"]
Oct 31 13:38:11.099662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:11.099621    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 13:38:11.099946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:11.099676    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb" podUID="b290805f-2d02-4d45-abbd-d0f0f8afe535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 13:38:11.332463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:11.332386    4428 patch_prober.go:29] interesting pod/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 31 13:38:11.332712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:11.332680    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x" podUID="c080a436-0943-4c33-938c-35b1ff2c1a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 31 13:38:11.413972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:11.413840    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" event={"ID":"d1af67f4-a0eb-402b-8453-7cb5ab463ab1","Type":"ContainerStarted","Data":"9fb715f96f38b48403dcf2b346071307d3e228d24b0cac0cc53101e386d544c5"}
Oct 31 13:38:11.413972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:11.413919    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" event={"ID":"d1af67f4-a0eb-402b-8453-7cb5ab463ab1","Type":"ContainerStarted","Data":"8852e6431856e545e349c6fd32ebf3c69f91057f7ed4b6d9cc67ab643c7321c6"}
Oct 31 13:38:11.526800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:11.518935    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" podStartSLOduration=3.518915943 podStartE2EDuration="3.518915943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:11.444660058 +0000 UTC m=+2088.843008686" watchObservedRunningTime="2026-10-31 13:38:11.518915943 +0000 UTC m=+2088.917264571"
Oct 31 13:38:11.715951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:11.700036    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:11.940181 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:11.940142    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:12.092579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:12.092478    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 13:38:12.092579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:12.092543    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb" podUID="b290805f-2d02-4d45-abbd-d0f0f8afe535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 13:38:12.645895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:12.636282    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:12.859896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:12.842217    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:38:12.859896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:12.851423    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:38:12.957616 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:12.950683    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:13.097965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:13.097934    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:38:13.099247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:13.099228    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 13:38:13.485910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:13.484985    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:13.565852 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:13.565806    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:13.585608 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:13.582341    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:38:14.043034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:14.042984    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:14.217632 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:14.217597    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:14.829184 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:14.829139    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:15.412290 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:15.412250    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:15.754980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:15.691323    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:15.942500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:15.942460    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:16.406034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:16.405479    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:16.732721 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:16.685473    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:16.842199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:16.835966    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:16.842199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:16.840066    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:16.971891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:16.971837    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:17.200672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:17.200550    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:17.232371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:17.232321    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:17.422892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:17.416161    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:17.662438 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:17.650779    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:17.852906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:17.844231    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:17.852906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:17.844606    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:38:17.852906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:17.845203    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:38:17.852906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:17.845273    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:38:17.875008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:17.868619    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:18.316897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:18.314969    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:18.384901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:18.367998    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:18.473895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:18.460949    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:38:18.479895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:18.478256    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:18.506756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:18.501978    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:18.542710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:18.519967    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:18.583480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:18.573824    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:38:18.817932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:18.817549    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:18.884172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:18.884071    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:38:19.044924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:19.018945    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:38:19.284895 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:38:19.234606    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:38:19.320900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:19.317268    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:19.320900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:19.318051    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:38:19.540561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:19.525959    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:38:19.542052 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:19.541324    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:38:19.543926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:19.543058    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:19.543926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:19.543493    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:38:19.583980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:19.583934    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:38:19.618895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:19.602034    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:38:19.661892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:19.627459    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:38:19.849115 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:19.763796    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:20.177451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:19.885778    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:38:20.201398 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:19.958837    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:20.241751 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:20.240801    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:20.485977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:20.470742    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:20.551336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:20.548987    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:20.551336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:20.550118    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:20.609905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:20.591645    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:38:20.640892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:20.627346    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:20.642604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:20.642513    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:20.714208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:20.714175    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:38:20.843589 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:20.843095    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:38:20.862259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:20.862231    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:38:20.944703 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:20.944635    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:38:21.149535 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:21.149399    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:21.331698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:21.331633    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:21.597587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:21.597554    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:21.943059 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:21.942965    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:22.164009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:22.156829    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:22.480928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:22.471194    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:22.891431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:22.890982    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:23.421981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:23.404819    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:23.546701 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:23.546142    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:23.602244 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:23.602182    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:23.819205 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:23.819156    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:23.897908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:23.836428    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Oct 31 13:38:23.937195 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:23.937138    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:23.949985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:23.949934    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:23.979801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:23.979768    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:38:23.998017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:23.978519    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:38:24.000530 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:24.000492    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:24.054578 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:24.054531    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:24.309566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:24.309515    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:24.417058 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:24.417022    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:38:24.597575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:24.597535    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:25.578336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:25.561557    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:26.330403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:26.330270    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:26.355516 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:26.355103    4428 patch_prober.go:29] interesting pod/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:38:26.355516 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:26.355170    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:27.559124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:27.558990    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:27.734483 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:27.734417    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:27.806064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:27.805998    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"]
Oct 31 13:38:27.848269 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:27.848168    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"]
Oct 31 13:38:27.848463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:27.848339    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:27.927124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:27.927081    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:27.927311 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:27.927169    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfg8b\" (UniqueName: \"kubernetes.io/projected/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-kube-api-access-bfg8b\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:27.927311 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:27.927239    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:27.927450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:27.927320    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:27.927450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:27.927377    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:27.927450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:27.927426    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:27.927599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:27.927544    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:27.927599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:27.927574    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:27.930838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:27.929161    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.033587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.033251    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.033587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.033343    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.033587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.033447    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.033587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.033590    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.033982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.033653    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfg8b\" (UniqueName: \"kubernetes.io/projected/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-kube-api-access-bfg8b\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.033982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.033766    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.033982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.033828    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.034191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.034167    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.034649 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.034621    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.034649 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.034640    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.034970 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.034682    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.034970 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.034716    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.035162 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.035069    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.035759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.035714    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.038139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.038112    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.038953 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.038928    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.051574 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.051504    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.054065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.054030    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfg8b\" (UniqueName: \"kubernetes.io/projected/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-kube-api-access-bfg8b\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.212093 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.212048    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:28.481010 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.480844    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:28.620693 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.620627    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"]
Oct 31 13:38:28.621215 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:38:28.619334    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b75287_cf60_49fe_8b06_dec5a12b5cd1.slice/crio-1023d7432271b97af10429eb4313b8bfac0a2fd62994c87450ea0f385fd5881f WatchSource:0}: Error finding container 1023d7432271b97af10429eb4313b8bfac0a2fd62994c87450ea0f385fd5881f: Status 404 returned error can't find the container with id 1023d7432271b97af10429eb4313b8bfac0a2fd62994c87450ea0f385fd5881f
Oct 31 13:38:28.623981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.623932    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:28.624120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.624044    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:28.624120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.624101    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:28.974833 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:28.974775    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:29.271008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:29.270963    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"]
Oct 31 13:38:29.279405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:29.279364    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cbb3d5-cff67"]
Oct 31 13:38:29.401820 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:29.401776    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:29.422357 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:29.422313    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n" event={"ID":"e0b75287-cf60-49fe-8b06-dec5a12b5cd1","Type":"ContainerStarted","Data":"d9479088ff99629a17781a69a3faf52cd9210756030aab30452ace9fc85aef59"}
Oct 31 13:38:29.422575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:29.422388    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n" event={"ID":"e0b75287-cf60-49fe-8b06-dec5a12b5cd1","Type":"ContainerStarted","Data":"1023d7432271b97af10429eb4313b8bfac0a2fd62994c87450ea0f385fd5881f"}
Oct 31 13:38:29.458949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:29.458806    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n" podStartSLOduration=2.458783874 podStartE2EDuration="2.458783874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:29.451572315 +0000 UTC m=+2106.849920976" watchObservedRunningTime="2026-10-31 13:38:29.458783874 +0000 UTC m=+2106.857132499"
Oct 31 13:38:29.599947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:29.592509    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:29.815780 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:29.815740    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:30.188428 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:30.188374    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:30.213046 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:30.212988    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:30.218484 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:30.218440    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 13:38:30.218694 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:30.218523    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n" podUID="e0b75287-cf60-49fe-8b06-dec5a12b5cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 13:38:30.811488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:30.811435    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="756a7df3-571b-4684-a369-01217173c597" path="/var/lib/kubelet/pods/756a7df3-571b-4684-a369-01217173c597/volumes"
Oct 31 13:38:30.877791 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:30.876741    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:31.182039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:31.164994    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:38:31.182039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:31.165077    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:31.213130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:31.213084    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 13:38:31.213347 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:31.213158    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n" podUID="e0b75287-cf60-49fe-8b06-dec5a12b5cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 13:38:32.229848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.214234    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 13:38:32.229848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.214315    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n" podUID="e0b75287-cf60-49fe-8b06-dec5a12b5cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 13:38:32.304946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.291341    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:32.624530 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.624496    4428 generic.go:356] "Generic (PLEG): container finished" podID="e6517163-656a-4391-a298-4c187010beae" containerID="ad4686d66ca13445f4e4dff2af2ef7904296be88a65e415bef969e3a3d22d10c" exitCode=0
Oct 31 13:38:32.624763 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.624575    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" event={"ID":"e6517163-656a-4391-a298-4c187010beae","Type":"ContainerDied","Data":"ad4686d66ca13445f4e4dff2af2ef7904296be88a65e415bef969e3a3d22d10c"}
Oct 31 13:38:32.625265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.625242    4428 scope.go:122] "RemoveContainer" containerID="ad4686d66ca13445f4e4dff2af2ef7904296be88a65e415bef969e3a3d22d10c"
Oct 31 13:38:32.627502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.627479    4428 generic.go:356] "Generic (PLEG): container finished" podID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerID="571c05265fb8792d5e55bf6d8a08b690d0efb1dfa84fafa7319e6ce33250ec7a" exitCode=0
Oct 31 13:38:32.627611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.627508    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" event={"ID":"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74","Type":"ContainerDied","Data":"571c05265fb8792d5e55bf6d8a08b690d0efb1dfa84fafa7319e6ce33250ec7a"}
Oct 31 13:38:32.628075 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.628031    4428 scope.go:122] "RemoveContainer" containerID="571c05265fb8792d5e55bf6d8a08b690d0efb1dfa84fafa7319e6ce33250ec7a"
Oct 31 13:38:32.635656 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.635524    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:32.635656 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.635630    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:32.635977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.635681    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:32.637016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.636986    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:32.637132 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.637068    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:32.637132 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.637111    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:32.794612 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.794584    4428 scope.go:122] "RemoveContainer" containerID="5942299c823b74cff32f19482b9cfd811eddf7d92a15554c8bba531cd724b9bc"
Oct 31 13:38:32.869287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.869258    4428 scope.go:122] "RemoveContainer" containerID="e6789be8ed37c1bca537cf625f62d723e6626ce5eeb8da53944a94bb5fbcc288"
Oct 31 13:38:32.890982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:32.890959    4428 scope.go:122] "RemoveContainer" containerID="b4bf13ff74b9c112e8a5f9684c984f4fb9089a7a0ef94130e3458e70f7368024"
Oct 31 13:38:33.213527 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:33.213493    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 13:38:33.213719 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:33.213563    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n" podUID="e0b75287-cf60-49fe-8b06-dec5a12b5cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 13:38:33.450414 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:33.450351    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:33.684160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:33.684046    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" event={"ID":"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74","Type":"ContainerStarted","Data":"04041c2d8e8caaea1b2dfbe3c3d2578a27ca932badca2417c101e56c6cfbea02"}
Oct 31 13:38:33.691893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:33.691840    4428 generic.go:356] "Generic (PLEG): container finished" podID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerID="1e26e91b65e5cb9ec7273df8e235af5f5b11d66e27279d7c5205d3e2cf060432" exitCode=0
Oct 31 13:38:33.692506 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:33.692479    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" event={"ID":"f0a9f039-bd18-4c3a-88a3-973bc6721086","Type":"ContainerDied","Data":"1e26e91b65e5cb9ec7273df8e235af5f5b11d66e27279d7c5205d3e2cf060432"}
Oct 31 13:38:33.696096 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:33.696072    4428 scope.go:122] "RemoveContainer" containerID="1e26e91b65e5cb9ec7273df8e235af5f5b11d66e27279d7c5205d3e2cf060432"
Oct 31 13:38:33.699837 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:33.699785    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:33.700160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:33.700130    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:33.700410 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:33.700226    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" event={"ID":"e6517163-656a-4391-a298-4c187010beae","Type":"ContainerStarted","Data":"ec70f8a05fea67eac9a8733e166c147282ea548d10bffd295ea779a20c5bd8a8"}
Oct 31 13:38:33.700510 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:33.700467    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:33.840083 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:33.840043    4428 patch_prober.go:29] interesting pod/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:38:33.840372 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:33.840340    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 31 13:38:33.840777 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:33.840759    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:38:34.212901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:34.185018    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:38:34.220611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:34.220039    4428 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:38:34.220611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:34.220123    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 31 13:38:34.220611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:34.220476    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 13:38:34.220611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:34.220514    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n" podUID="e0b75287-cf60-49fe-8b06-dec5a12b5cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 13:38:34.331131 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:34.331087    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:34.901896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:34.878992    4428 patch_prober.go:29] interesting pod/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:38:34.901896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:34.879059    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 31 13:38:35.082429 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:35.082377    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" event={"ID":"f0a9f039-bd18-4c3a-88a3-973bc6721086","Type":"ContainerStarted","Data":"5772b0da2ef97500dc3b575bed722d9e518fb240c624309f6e14b9c5dbba38c8"}
Oct 31 13:38:35.166958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:35.161313    4428 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:38:35.166958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:35.161373    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 31 13:38:35.223964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:35.223387    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 13:38:35.223964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:35.223463    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n" podUID="e0b75287-cf60-49fe-8b06-dec5a12b5cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 13:38:35.363662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:35.363501    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:35.812895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:35.812734    4428 patch_prober.go:29] interesting pod/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:38:35.812895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:35.812797    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 31 13:38:35.987895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:35.987447    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:38:35.987895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:35.987575    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 13:38:35.987895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:35.987651    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 13:38:36.167367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:36.167268    4428 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:38:36.167638 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:36.167606    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 31 13:38:36.215555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:36.215516    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 13:38:36.226940 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:36.215834    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n" podUID="e0b75287-cf60-49fe-8b06-dec5a12b5cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 13:38:36.804991 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:36.792001    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:37.245021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:36.792325    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:37.246266 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:36.841405    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:37.282928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:36.902601    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:38:37.283457 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:36.956961    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:37.283792 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:36.958576    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:37.284124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:36.997329    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 13:38:37.290891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:37.284259    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 13:38:37.319909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:36.997771    4428 patch_prober.go:29] interesting pod/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:38:37.319909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:37.300504    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 31 13:38:37.319909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:37.241907    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:38:37.319909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:37.186524    4428 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:38:37.319909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:37.300821    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 31 13:38:37.319909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:37.260314    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 13:38:37.319909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:37.300888    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n" podUID="e0b75287-cf60-49fe-8b06-dec5a12b5cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 13:38:37.486895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:37.470842    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:37.630223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:37.629654    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:38:37.867895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:37.819284    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"]
Oct 31 13:38:38.065023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:37.840001    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:38.151902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:37.841398    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:38.162340 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.069354    4428 patch_prober.go:29] interesting pod/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:38:38.188036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.187970    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 31 13:38:38.272924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.144552    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 13:38:38.384500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.384391    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 13:38:38.385000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.281652    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:38:38.387196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.257312    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:38.411893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.393741    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:38.429505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.335980    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:38.461180 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.336120    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:38:38.462204 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.351275    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded"
Oct 31 13:38:38.462462 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.363655    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:38.506905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.481102    4428 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:38:39.051523 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.051414    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 31 13:38:39.052316 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.363598    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:39.129894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.363686    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded"
Oct 31 13:38:39.129894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.336225    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:38:39.129894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.542845    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:38:39.129894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.674375    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0d835b-openshift-default-dockercfg-68mcz\""
Oct 31 13:38:39.129894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.780206    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-token\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.129894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.090244    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-podinfo\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.129894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.090340    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-workload-socket\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.129894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.090433    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-envoy\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.129894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.090518    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-workload-certs\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.129894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.090687    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-data\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.129894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.090768    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-credential-socket\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.129894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.090820    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx66n\" (UniqueName: \"kubernetes.io/projected/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-kube-api-access-lx66n\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.129894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.090957    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istiod-ca-cert\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.129894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.864572    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:39.170902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.819213    4428 patch_prober.go:29] interesting pod/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:38:39.170902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.166242    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 31 13:38:39.170902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.009704    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 13:38:39.170902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.166355    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 13:38:39.170902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.009770    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Oct 31 13:38:39.170902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:38.524663    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:38:39.170902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.166679    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:38:39.170902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.166790    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:38:39.170902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.166817    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"]
Oct 31 13:38:39.185480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.180650    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:38:39.185480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.180722    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:38:39.185480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.180777    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:38:39.288895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.261636    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-data\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.288895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.261733    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-credential-socket\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.288895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.261783    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx66n\" (UniqueName: \"kubernetes.io/projected/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-kube-api-access-lx66n\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.322580 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.322443    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-data\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.330050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.330008    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:38:39.383790 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.354367    4428 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:38:39.384702 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.384645    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 31 13:38:39.431117 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.431073    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:39.465588 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.465550    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-credential-socket\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.466578 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.466550    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istiod-ca-cert\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.468083 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.468018    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:39.468544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.468516    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:39.534109 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.534051    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:39.603008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.586028    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-token\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.603008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.586256    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-podinfo\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.603008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.586346    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-workload-socket\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.603008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.586452    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-envoy\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.603008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.586549    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-workload-certs\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.765017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.698988    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:39.765017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.731061    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istiod-ca-cert\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:39.937209 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.796975    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:38:39.937209 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.797039    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n" podUID="e0b75287-cf60-49fe-8b06-dec5a12b5cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:39.958893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.945018    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-workload-socket\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:40.019889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.986088    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-token\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:40.019889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.986529    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-podinfo\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:40.019889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:39.987694    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-workload-certs\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:40.019889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.018330    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:40.052004 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.051970    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx66n\" (UniqueName: \"kubernetes.io/projected/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-kube-api-access-lx66n\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:40.055174 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.055129    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:40.070921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.057535    4428 patch_prober.go:29] interesting pod/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:38:40.136177 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.131462    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 31 13:38:40.145919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.116998    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:40.240951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.196887    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:40.240951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.197171    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 13:38:40.240951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.215697    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 13:38:40.240951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.221811    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:40.430219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.410157    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:38:40.438740 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.438703    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-envoy\") pod \"e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") " pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:40.477847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.469527    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:40.539163 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:38:40.539075    4428 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.358s"
Oct 31 13:38:40.539539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.539500    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:40.561777 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.561749    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:38:40.562167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.562151    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:38:40.591731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.591704    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"]
Oct 31 13:38:40.616055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.615991    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:40.617672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.617637    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:38:40.664925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.588372    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:40.665431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.665299    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:38:40.666497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.666452    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:38:40.687897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.672941    4428 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:38:40.687897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:40.672994    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 31 13:38:41.032630 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.032584    4428 patch_prober.go:29] interesting pod/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:38:41.098973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.080534    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 31 13:38:41.152893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.139127    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:38:41.171563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.171530    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cae9ec-serving-cert\""
Oct 31 13:38:41.231895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.208424    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 13:38:41.231895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.208480    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 13:38:41.231895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.208978    4428 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:38:41.231895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.209021    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 31 13:38:41.231895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.210485    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:41.231895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.224978    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:41.231895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.226550    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:38:41.248681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.248641    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:41.354518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.347578    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/545cc90a-0eda-4923-9757-c351ccac5d73-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk\" (UID: \"545cc90a-0eda-4923-9757-c351ccac5d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:38:41.355076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.354324    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:41.356085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.356048    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf7pz\" (UniqueName: \"kubernetes.io/projected/545cc90a-0eda-4923-9757-c351ccac5d73-kube-api-access-vf7pz\") pod \"maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk\" (UID: \"545cc90a-0eda-4923-9757-c351ccac5d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:38:41.356356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.356338    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/545cc90a-0eda-4923-9757-c351ccac5d73-metrics-certs\") pod \"maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk\" (UID: \"545cc90a-0eda-4923-9757-c351ccac5d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:38:41.378581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.378533    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/545cc90a-0eda-4923-9757-c351ccac5d73-maas-api-tls\") pod \"maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk\" (UID: \"545cc90a-0eda-4923-9757-c351ccac5d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:38:41.440982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.412820    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:41.497287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.496212    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/545cc90a-0eda-4923-9757-c351ccac5d73-maas-api-tls\") pod \"maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk\" (UID: \"545cc90a-0eda-4923-9757-c351ccac5d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:38:41.497287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.496459    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/545cc90a-0eda-4923-9757-c351ccac5d73-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk\" (UID: \"545cc90a-0eda-4923-9757-c351ccac5d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:38:41.497287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.496539    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf7pz\" (UniqueName: \"kubernetes.io/projected/545cc90a-0eda-4923-9757-c351ccac5d73-kube-api-access-vf7pz\") pod \"maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk\" (UID: \"545cc90a-0eda-4923-9757-c351ccac5d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:38:41.497287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.496586    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/545cc90a-0eda-4923-9757-c351ccac5d73-metrics-certs\") pod \"maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk\" (UID: \"545cc90a-0eda-4923-9757-c351ccac5d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:38:41.554400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.520382    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/545cc90a-0eda-4923-9757-c351ccac5d73-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk\" (UID: \"545cc90a-0eda-4923-9757-c351ccac5d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:38:41.554400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.534285    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/545cc90a-0eda-4923-9757-c351ccac5d73-maas-api-tls\") pod \"maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk\" (UID: \"545cc90a-0eda-4923-9757-c351ccac5d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:38:41.574935 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.567074    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:41.665966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.623431    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf7pz\" (UniqueName: \"kubernetes.io/projected/545cc90a-0eda-4923-9757-c351ccac5d73-kube-api-access-vf7pz\") pod \"maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk\" (UID: \"545cc90a-0eda-4923-9757-c351ccac5d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:38:41.733900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.693056    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:38:41.733900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.693156    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:38:41.736899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.693182    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:38:41.737596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.693201    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:38:41.747945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.747913    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"]
Oct 31 13:38:41.748238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.748218    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:41.748445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.748432    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:38:41.748578 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.748566    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:38:41.748762 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.748748    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:38:41.758340 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.757774    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:38:41.768912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.758584    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:38:41.814063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.802910    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 13:38:41.871105 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.844384    4428 patch_prober.go:29] interesting pod/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:38:41.871105 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.844442    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 31 13:38:42.119910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:41.982538    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 13:38:42.481433 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:42.481387    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 13:38:42.490054 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:42.158278    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:38:42.596896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:42.592957    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"]
Oct 31 13:38:42.729968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:42.628051    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:38:42.729968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:42.032452    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:42.729968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:42.166015    4428 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:38:42.729968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:42.628819    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 31 13:38:42.729968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:42.629018    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66" podUID="73e0f5e0-d08d-4acd-8315-e3fe0ad038c2" containerName="maas-api" containerID="cri-o://e2eb6288b90f0620b7bf829340c7475019d26104cdb8a12d15943470ca78b91c" gracePeriod=30
Oct 31 13:38:42.729968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:42.678477    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/545cc90a-0eda-4923-9757-c351ccac5d73-metrics-certs\") pod \"maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk\" (UID: \"545cc90a-0eda-4923-9757-c351ccac5d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:38:42.923521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:42.923482    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:38:43.219051 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.209020    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:38:43.411903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.227119    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:38:43.431964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.326320    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:43.431964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.427085    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:43.431964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.427207    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:43.439135 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.350168    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 13:38:43.439425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.439395    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 13:38:43.439691 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.439667    4428 patch_prober.go:29] interesting pod/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:38:43.439837 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.439817    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 31 13:38:43.540436 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.540399    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:43.688128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.676326    4428 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:38:43.688128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.676420    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 31 13:38:43.856586 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.820450    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded"
Oct 31 13:38:43.858016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.857975    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:43.860023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.859858    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:43.869830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.869785    4428 patch_prober.go:29] interesting pod/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:38:43.870144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.870104    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 31 13:38:43.890271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.890224    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:43.891660 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:43.891367    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66" podUID="73e0f5e0-d08d-4acd-8315-e3fe0ad038c2" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.161:8443/health\": dial tcp 10.134.0.161:8443: connect: connection refused"
Oct 31 13:38:44.050893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:44.035144    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 13:38:44.050893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:44.035233    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 13:38:44.196992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:44.193620    4428 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Oct 31 13:38:44.197381 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:44.197343    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Oct 31 13:38:44.773661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:44.749199    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 13:38:44.983014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:44.936454    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:45.170612 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.063377    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:38:45.170612 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.073896    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:38:45.210899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.182855    4428 patch_prober.go:29] interesting pod/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 31 13:38:45.367523 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.367468    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 31 13:38:45.367817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.268009    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:38:45.369464 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.274117    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 13:38:45.369694 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.369666    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 13:38:45.413426 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.412387    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:45.433682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.418284    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s" podUID="371c7f96-669c-455b-a063-29303ee30cb2" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 31 13:38:45.433682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.429754    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 13:38:45.443215 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.440006    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hql47" podUID="ea978050-20b0-4148-ab41-75844fe1b637" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:45.482550 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.479562    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:45.516433 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:38:45.515669    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e0f5e0_d08d_4acd_8315_e3fe0ad038c2.slice/crio-e2eb6288b90f0620b7bf829340c7475019d26104cdb8a12d15943470ca78b91c.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 13:38:45.518566 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:38:45.518531    4428 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.291s"
Oct 31 13:38:45.518734 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.518633    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:38:45.594114 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.594086    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:38:45.594409 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.594311    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:38:45.594522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.594503    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp" event={"ID":"ace3362d-9bb7-46e2-b218-4e7d3e0ed354","Type":"ContainerStarted","Data":"0e7a39020deceac4aaa5605ca274062fc654a2415fadc829492dc15274870ad1"}
Oct 31 13:38:45.594664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.594651    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:38:45.594818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.594805    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:38:45.595104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.595088    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:38:45.595268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.595253    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"]
Oct 31 13:38:45.596570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.596532    4428 generic.go:356] "Generic (PLEG): container finished" podID="73e0f5e0-d08d-4acd-8315-e3fe0ad038c2" containerID="e2eb6288b90f0620b7bf829340c7475019d26104cdb8a12d15943470ca78b91c" exitCode=0
Oct 31 13:38:45.605008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.604988    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"]
Oct 31 13:38:45.605222 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.605203    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66" event={"ID":"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2","Type":"ContainerDied","Data":"e2eb6288b90f0620b7bf829340c7475019d26104cdb8a12d15943470ca78b91c"}
Oct 31 13:38:45.605424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.605387    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:38:45.607475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.605581    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.608427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.606600    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:38:45.608607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.608597    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:38:45.609167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.609154    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:38:45.609604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.609593    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:38:45.625614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.625561    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"]
Oct 31 13:38:45.704464 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.704285    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1ddade-openshift-default-dockercfg-vl7sd\""
Oct 31 13:38:45.728889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.727096    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-credential-socket\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.728889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.727220    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-istio-envoy\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.728889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.727599    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6349679a-67f7-41e5-b289-dc91302b5977-istio-podinfo\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.728889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.727991    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwshd\" (UniqueName: \"kubernetes.io/projected/6349679a-67f7-41e5-b289-dc91302b5977-kube-api-access-mwshd\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.728889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.728060    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6349679a-67f7-41e5-b289-dc91302b5977-istio-token\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.728889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.728271    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-workload-socket\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.728889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.728368    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6349679a-67f7-41e5-b289-dc91302b5977-istiod-ca-cert\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.728889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.728509    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-workload-certs\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.728889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.728620    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-istio-data\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.743904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.734242    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:45.743904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.739584    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"]
Oct 31 13:38:45.824264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.824239    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:38:45.827087 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.827068    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:38:45.830882 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.830838    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6349679a-67f7-41e5-b289-dc91302b5977-istio-token\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.831483 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.831461    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-workload-socket\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.831672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.831656    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6349679a-67f7-41e5-b289-dc91302b5977-istiod-ca-cert\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.832083 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.832060    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-workload-certs\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.832196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.832129    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-istio-data\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.832265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.832230    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-credential-socket\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.832323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.832289    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-istio-envoy\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.832819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.832757    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-istio-data\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.832819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.831928    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-workload-socket\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.832819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.832780    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-workload-certs\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.833819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.833212    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6349679a-67f7-41e5-b289-dc91302b5977-istio-podinfo\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.833819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.833316    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-credential-socket\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.833819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.833390    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwshd\" (UniqueName: \"kubernetes.io/projected/6349679a-67f7-41e5-b289-dc91302b5977-kube-api-access-mwshd\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.835023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.835002    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6349679a-67f7-41e5-b289-dc91302b5977-istiod-ca-cert\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.843149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.843120    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6349679a-67f7-41e5-b289-dc91302b5977-istio-podinfo\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.844672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.844650    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-istio-envoy\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.895910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.892624    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6349679a-67f7-41e5-b289-dc91302b5977-istio-token\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.899934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.899901    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwshd\" (UniqueName: \"kubernetes.io/projected/6349679a-67f7-41e5-b289-dc91302b5977-kube-api-access-mwshd\") pod \"e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") " pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:45.907231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.907195    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:45.923636 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:45.923602    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:38:46.006906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:46.003120    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 13:38:46.006906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:46.003189    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 13:38:46.006906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:46.005692    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:46.044571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:46.036296    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-openshift-service-ca\") pod \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\" (UID: \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\") "
Oct 31 13:38:46.044571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:46.036578    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-metrics-certs\") pod \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\" (UID: \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\") "
Oct 31 13:38:46.044571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:46.036612    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-maas-api-tls\") pod \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\" (UID: \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\") "
Oct 31 13:38:46.044571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:46.036755    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-kube-api-access-c64vc\" (UniqueName: \"kubernetes.io/projected/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-kube-api-access-c64vc\") pod \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\" (UID: \"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2\") "
Oct 31 13:38:46.130353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:46.083215    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-openshift-service-ca" pod "73e0f5e0-d08d-4acd-8315-e3fe0ad038c2" (UID: "73e0f5e0-d08d-4acd-8315-e3fe0ad038c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:38:46.331908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:46.313643    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:38:46.331908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:46.331560    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-metrics-certs" pod "73e0f5e0-d08d-4acd-8315-e3fe0ad038c2" (UID: "73e0f5e0-d08d-4acd-8315-e3fe0ad038c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:38:46.349012 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:46.348953    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-kube-api-access-c64vc" pod "73e0f5e0-d08d-4acd-8315-e3fe0ad038c2" (UID: "73e0f5e0-d08d-4acd-8315-e3fe0ad038c2"). InnerVolumeSpecName "kube-api-access-c64vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:38:46.431992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:46.417098    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:38:46.431992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:46.417138    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c64vc\" (UniqueName: \"kubernetes.io/projected/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-kube-api-access-c64vc\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:38:46.777988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:46.776315    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-maas-api-tls" pod "73e0f5e0-d08d-4acd-8315-e3fe0ad038c2" (UID: "73e0f5e0-d08d-4acd-8315-e3fe0ad038c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:38:46.880634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:46.866087    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:38:47.012946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.004688    4428 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 13:38:47.012946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.004748    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 13:38:47.012946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.010637    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:47.244152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.241153    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk" event={"ID":"545cc90a-0eda-4923-9757-c351ccac5d73","Type":"ContainerStarted","Data":"e547f1394423797351e49bc90460b80f8deac6c54d924cdf2be0d1fe60b9ed31"}
Oct 31 13:38:47.263013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.250820    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66" event={"ID":"73e0f5e0-d08d-4acd-8315-e3fe0ad038c2","Type":"ContainerDied","Data":"2823dca9b71de988235fd4f1057ac47e212b386dbdded1e6f5203d1b30e89b9c"}
Oct 31 13:38:47.263013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.250918    4428 scope.go:122] "RemoveContainer" containerID="e2eb6288b90f0620b7bf829340c7475019d26104cdb8a12d15943470ca78b91c"
Oct 31 13:38:47.263013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.251158    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"
Oct 31 13:38:47.263013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.261673    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp" event={"ID":"ace3362d-9bb7-46e2-b218-4e7d3e0ed354","Type":"ContainerStarted","Data":"2a73b302ef8ff6424a483a0adfe6e5967a7ec5a467db474c3b2bcd12b1ad5517"}
Oct 31 13:38:47.317688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.315954    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"]
Oct 31 13:38:47.357564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.357526    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c45a37-58fdbf57fc-84t66"]
Oct 31 13:38:47.404803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.404723    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp" podStartSLOduration=10.404699835 podStartE2EDuration="10.404699835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:47.339345578 +0000 UTC m=+2124.737694264" watchObservedRunningTime="2026-10-31 13:38:47.404699835 +0000 UTC m=+2124.803048463"
Oct 31 13:38:47.612962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.602530    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:47.645944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.642425    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:47.646359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.646322    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:47.646542 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.646522    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:38:47.684642 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.684614    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:47.718691 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.718655    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"]
Oct 31 13:38:47.892476 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.892443    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"]
Oct 31 13:38:47.893985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.893956    4428 cpu_manager.go:414] "containerMap: removing container" podUID="73e0f5e0-d08d-4acd-8315-e3fe0ad038c2" containerName="maas-api"
Oct 31 13:38:47.893985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.893982    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e0f5e0-d08d-4acd-8315-e3fe0ad038c2" containerName="maas-api"
Oct 31 13:38:47.894240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.894222    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e0f5e0-d08d-4acd-8315-e3fe0ad038c2" containerName="maas-api"
Oct 31 13:38:47.916670 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.899538    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:38:47.916670 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.904848    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4fde10-serving-cert\""
Oct 31 13:38:47.920887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.920819    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"]
Oct 31 13:38:47.991622 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.989705    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:38:47.991622 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.991414    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:38:47.998618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.998586    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0031a9-bab3-4590-8add-4eeafdf4ac52-maas-api-tls\") pod \"maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk\" (UID: \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:38:47.998773 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.998646    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0031a9-bab3-4590-8add-4eeafdf4ac52-metrics-certs\") pod \"maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk\" (UID: \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:38:47.998773 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.998734    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk2r5\" (UniqueName: \"kubernetes.io/projected/1e0031a9-bab3-4590-8add-4eeafdf4ac52-kube-api-access-fk2r5\") pod \"maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk\" (UID: \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:38:47.999066 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:47.999045    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0031a9-bab3-4590-8add-4eeafdf4ac52-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk\" (UID: \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:38:48.002976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.002949    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:48.103906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.101926    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0031a9-bab3-4590-8add-4eeafdf4ac52-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk\" (UID: \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:38:48.103906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.102147    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0031a9-bab3-4590-8add-4eeafdf4ac52-maas-api-tls\") pod \"maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk\" (UID: \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:38:48.103906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.102205    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0031a9-bab3-4590-8add-4eeafdf4ac52-metrics-certs\") pod \"maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk\" (UID: \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:38:48.103906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.102260    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk2r5\" (UniqueName: \"kubernetes.io/projected/1e0031a9-bab3-4590-8add-4eeafdf4ac52-kube-api-access-fk2r5\") pod \"maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk\" (UID: \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:38:48.103906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.103632    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0031a9-bab3-4590-8add-4eeafdf4ac52-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk\" (UID: \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:38:48.137292 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.137195    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0031a9-bab3-4590-8add-4eeafdf4ac52-maas-api-tls\") pod \"maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk\" (UID: \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:38:48.194915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.173975    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:48.194915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.179307    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0031a9-bab3-4590-8add-4eeafdf4ac52-metrics-certs\") pod \"maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk\" (UID: \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:38:48.194915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.181575    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk2r5\" (UniqueName: \"kubernetes.io/projected/1e0031a9-bab3-4590-8add-4eeafdf4ac52-kube-api-access-fk2r5\") pod \"maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk\" (UID: \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:38:48.244280 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.233817    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:38:48.537981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.536583    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk" event={"ID":"545cc90a-0eda-4923-9757-c351ccac5d73","Type":"ContainerStarted","Data":"b66d28ec0ae214f95041df445e37e1658eecd2ed5974576181af74076a3f69ac"}
Oct 31 13:38:48.553178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.539304    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:38:48.553178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.542196    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx" event={"ID":"6349679a-67f7-41e5-b289-dc91302b5977","Type":"ContainerStarted","Data":"ef28b27b58d8d70c68447f988100a8adc7d58ec0c399ab24db584a2f8a845c7f"}
Oct 31 13:38:48.553178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.542256    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx" event={"ID":"6349679a-67f7-41e5-b289-dc91302b5977","Type":"ContainerStarted","Data":"edb13254c9d78a03be65316926620419999b08a8e3a7a24b8a70841884c6e0e8"}
Oct 31 13:38:48.653955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.640881    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk" podStartSLOduration=9.493106593 podStartE2EDuration="10.640847532s" totalImagesPullingTime="1.147740939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:48.592969075 +0000 UTC m=+2125.991317700" watchObservedRunningTime="2026-10-31 13:38:48.640847532 +0000 UTC m=+2126.039196350"
Oct 31 13:38:48.653955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.644590    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx" podStartSLOduration=9.644574768 podStartE2EDuration="9.644574768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:48.625760837 +0000 UTC m=+2126.024109463" watchObservedRunningTime="2026-10-31 13:38:48.644574768 +0000 UTC m=+2126.042923438"
Oct 31 13:38:48.676009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.675945    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:48.676227 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:48.676126    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:38:49.316370 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:38:49.316274    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:38:49.340072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:49.337468    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e0f5e0-d08d-4acd-8315-e3fe0ad038c2" path="/var/lib/kubelet/pods/73e0f5e0-d08d-4acd-8315-e3fe0ad038c2/volumes"
Oct 31 13:38:49.340072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:49.338038    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:49.340072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:49.338146    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:49.398750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:49.398683    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"]
Oct 31 13:38:49.403792 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:49.403761    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f518f2-24nd7"]
Oct 31 13:38:49.409636 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:49.409606    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"]
Oct 31 13:38:49.529440 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:49.529371    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:49.743187 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:49.743134    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk" event={"ID":"1e0031a9-bab3-4590-8add-4eeafdf4ac52","Type":"ContainerStarted","Data":"3e89206053d6b87efd82097b8dcd2301cc467c5b8e6deeb51e95c2d9f9b70d3b"}
Oct 31 13:38:49.744032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:49.744002    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:49.748897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:49.747088    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:38:49.824772 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:49.824738    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:49.991985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:49.991953    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:50.672893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:50.661469    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:50.800750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:50.800714    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b20618-6142-430f-bfc5-108a93cb3b54" path="/var/lib/kubelet/pods/10b20618-6142-430f-bfc5-108a93cb3b54/volumes"
Oct 31 13:38:50.801410 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:50.801387    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk" event={"ID":"1e0031a9-bab3-4590-8add-4eeafdf4ac52","Type":"ContainerStarted","Data":"2db241a630f4956776f1179fb7d539350154dafeb4d37da4a750a2e0ebc78b26"}
Oct 31 13:38:50.822842 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:50.822783    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk" podStartSLOduration=3.269464365 podStartE2EDuration="3.822768185s" totalImagesPullingTime="553.30382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:50.820398212 +0000 UTC m=+2128.218746840" watchObservedRunningTime="2026-10-31 13:38:50.822768185 +0000 UTC m=+2128.221116806"
Oct 31 13:38:51.198961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:51.178442    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:51.606891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:51.598779    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:51.892231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:51.891942    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:52.149525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:52.149427    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:52.387922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:52.360051    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:52.487743 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:52.485511    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:52.532767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:52.494607    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:52.596635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:52.591751    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:38:52.703639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:52.703599    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:52.741720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:52.701851    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:53.245570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:53.241964    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:53.438161 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:53.438116    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:53.487510 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:53.485922    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:38:53.487510 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:53.486123    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:38:53.507176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:53.507088    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:38:53.618615 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:53.618519    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:53.876043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:53.871377    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:54.331893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:54.322130    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:54.596317 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:54.596233    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:54.608805 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:54.608775    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"]
Oct 31 13:38:54.615272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:54.615211    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"
Oct 31 13:38:54.625858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:54.625725    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1ddade\""
Oct 31 13:38:54.634497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:54.634469    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"]
Oct 31 13:38:54.648909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:54.647464    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1ddade-dockercfg-xww8d\""
Oct 31 13:38:54.718996 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:54.710541    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjvgl\" (UniqueName: \"kubernetes.io/projected/42ac93e6-9ed8-4aee-b19d-3e55fbd4170f-kube-api-access-tjvgl\") pod \"payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7\" (UID: \"42ac93e6-9ed8-4aee-b19d-3e55fbd4170f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"
Oct 31 13:38:54.720826 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:54.720793    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ac93e6-9ed8-4aee-b19d-3e55fbd4170f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7\" (UID: \"42ac93e6-9ed8-4aee-b19d-3e55fbd4170f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"
Oct 31 13:38:54.838964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:54.824919    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:54.871951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:54.870157    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjvgl\" (UniqueName: \"kubernetes.io/projected/42ac93e6-9ed8-4aee-b19d-3e55fbd4170f-kube-api-access-tjvgl\") pod \"payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7\" (UID: \"42ac93e6-9ed8-4aee-b19d-3e55fbd4170f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"
Oct 31 13:38:54.871951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:54.870409    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ac93e6-9ed8-4aee-b19d-3e55fbd4170f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7\" (UID: \"42ac93e6-9ed8-4aee-b19d-3e55fbd4170f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"
Oct 31 13:38:54.875888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:54.872484    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ac93e6-9ed8-4aee-b19d-3e55fbd4170f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7\" (UID: \"42ac93e6-9ed8-4aee-b19d-3e55fbd4170f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"
Oct 31 13:38:55.068578 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:55.068474    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjvgl\" (UniqueName: \"kubernetes.io/projected/42ac93e6-9ed8-4aee-b19d-3e55fbd4170f-kube-api-access-tjvgl\") pod \"payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7\" (UID: \"42ac93e6-9ed8-4aee-b19d-3e55fbd4170f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"
Oct 31 13:38:55.264554 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:55.264263    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"
Oct 31 13:38:55.757158 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:55.747351    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:55.794238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:55.793820    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"]
Oct 31 13:38:55.811191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:55.810007    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:38:55.812514 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:55.812473    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7" event={"ID":"42ac93e6-9ed8-4aee-b19d-3e55fbd4170f","Type":"ContainerStarted","Data":"5895abaddf85877753a0406820ad43ab79ebdea5bf1e860a011394f2e7f044ca"}
Oct 31 13:38:56.721941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:56.719750    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:56.845289 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:56.845239    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7" event={"ID":"42ac93e6-9ed8-4aee-b19d-3e55fbd4170f","Type":"ContainerStarted","Data":"867716c344290a0bd78a7b3e8bcbd8a5496eadecd397512c5dddbff2cc52c892"}
Oct 31 13:38:56.845774 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:56.845532    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"
Oct 31 13:38:56.870971 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:56.870909    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7" podStartSLOduration=2.870889642 podStartE2EDuration="2.870889642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:56.866146421 +0000 UTC m=+2134.264495045" watchObservedRunningTime="2026-10-31 13:38:56.870889642 +0000 UTC m=+2134.269238267"
Oct 31 13:38:57.484218 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:57.484175    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:57.661022 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:57.660985    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:58.272302 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:58.272251    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:58.612805 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:58.612719    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:38:58.810508 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:58.810470    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:58.825066 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:58.825031    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"]
Oct 31 13:38:58.830134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:58.830107    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"
Oct 31 13:38:58.838047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:58.837822    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-554759-dockercfg-7wrzb\""
Oct 31 13:38:58.849043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:58.849010    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-554759\""
Oct 31 13:38:58.870264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:58.870180    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"]
Oct 31 13:38:59.080977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:59.023513    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:59.083728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:59.056998    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb630d97-8786-4349-b029-96ed7eda31a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd\" (UID: \"eb630d97-8786-4349-b029-96ed7eda31a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"
Oct 31 13:38:59.106724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:59.106519    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4znz\" (UniqueName: \"kubernetes.io/projected/eb630d97-8786-4349-b029-96ed7eda31a5-kube-api-access-b4znz\") pod \"payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd\" (UID: \"eb630d97-8786-4349-b029-96ed7eda31a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"
Oct 31 13:38:59.248902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:59.246163    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb630d97-8786-4349-b029-96ed7eda31a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd\" (UID: \"eb630d97-8786-4349-b029-96ed7eda31a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"
Oct 31 13:38:59.248902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:59.246335    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4znz\" (UniqueName: \"kubernetes.io/projected/eb630d97-8786-4349-b029-96ed7eda31a5-kube-api-access-b4znz\") pod \"payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd\" (UID: \"eb630d97-8786-4349-b029-96ed7eda31a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"
Oct 31 13:38:59.248902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:59.247679    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb630d97-8786-4349-b029-96ed7eda31a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd\" (UID: \"eb630d97-8786-4349-b029-96ed7eda31a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"
Oct 31 13:38:59.325950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:59.312163    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4znz\" (UniqueName: \"kubernetes.io/projected/eb630d97-8786-4349-b029-96ed7eda31a5-kube-api-access-b4znz\") pod \"payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd\" (UID: \"eb630d97-8786-4349-b029-96ed7eda31a5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"
Oct 31 13:38:59.494347 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:59.494317    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"
Oct 31 13:38:59.813579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:59.813545    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"]
Oct 31 13:38:59.815103 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:38:59.815059    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb630d97_8786_4349_b029_96ed7eda31a5.slice/crio-b2dcffbbd2c2e18ebda87e70d983fb5deb1f4adbfaf0ae048da1273801327009 WatchSource:0}: Error finding container b2dcffbbd2c2e18ebda87e70d983fb5deb1f4adbfaf0ae048da1273801327009: Status 404 returned error can't find the container with id b2dcffbbd2c2e18ebda87e70d983fb5deb1f4adbfaf0ae048da1273801327009
Oct 31 13:38:59.894636 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:59.894590    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:38:59.983826 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:38:59.983785    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd" event={"ID":"eb630d97-8786-4349-b029-96ed7eda31a5","Type":"ContainerStarted","Data":"b2dcffbbd2c2e18ebda87e70d983fb5deb1f4adbfaf0ae048da1273801327009"}
Oct 31 13:39:00.194127 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:00.194031    4428 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:39:00.194127 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:00.194097    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww" podUID="d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:39:00.995062 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:00.995027    4428 generic.go:356] "Generic (PLEG): container finished" podID="d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" containerID="65a8f3089a0bf9247fc8fa15579daf487383d3c6d94707994105b32bc9df7881" exitCode=0
Oct 31 13:39:00.995628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:00.995104    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww" event={"ID":"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954","Type":"ContainerDied","Data":"65a8f3089a0bf9247fc8fa15579daf487383d3c6d94707994105b32bc9df7881"}
Oct 31 13:39:00.995628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:00.995619    4428 scope.go:122] "RemoveContainer" containerID="65a8f3089a0bf9247fc8fa15579daf487383d3c6d94707994105b32bc9df7881"
Oct 31 13:39:00.997155 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:00.997116    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd" event={"ID":"eb630d97-8786-4349-b029-96ed7eda31a5","Type":"ContainerStarted","Data":"814ab9606c651e4a556e2b516103aef26e2a999f1cc6c0d84d2a3b3b48a25e9f"}
Oct 31 13:39:00.997286 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:00.997262    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"
Oct 31 13:39:00.997819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:00.997792    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:39:00.997938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:00.997901    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:39:00.998013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:00.997950    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:39:01.049066 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:01.049004    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd" podStartSLOduration=3.048982399 podStartE2EDuration="3.048982399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:39:01.046093182 +0000 UTC m=+2138.444441803" watchObservedRunningTime="2026-10-31 13:39:01.048982399 +0000 UTC m=+2138.447331022"
Oct 31 13:39:01.230905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:01.225512    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:02.008041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:02.007999    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww" event={"ID":"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954","Type":"ContainerStarted","Data":"903a102639cbd6fffd6ee7f723cb568f6dc40343b4bc6f4b31fbce3da61165e6"}
Oct 31 13:39:02.557899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:02.557836    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:02.563520 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:02.563443    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:39:02.564141 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:02.564108    4428 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 13:39:02.564465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:02.564432    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww" podUID="d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 13:39:03.570404 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:03.570363    4428 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 13:39:03.571165 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:03.570446    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww" podUID="d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 13:39:03.963368 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:03.963332    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:04.569954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:04.568351    4428 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 13:39:04.569954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:04.568418    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww" podUID="d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 13:39:05.078628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:05.078574    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:05.565344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:05.565304    4428 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 13:39:05.565657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:05.565627    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww" podUID="d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 13:39:06.216904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:06.210674    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:06.565446 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:06.565338    4428 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 13:39:06.565747 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:06.565706    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww" podUID="d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 13:39:07.181894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:07.181808    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:07.596146 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:07.596112    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:39:07.596927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:07.596827    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:39:07.598401 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:07.598359    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:39:07.863832 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:07.863736    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"
Oct 31 13:39:08.183789 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:08.181542    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:08.921058 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:08.921025    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:10.044826 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:10.044129    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:10.944483 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:10.944441    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:11.704530 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:11.704480    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:12.011403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:12.011371    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"
Oct 31 13:39:12.487579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:12.481903    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:13.324832 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:13.324791    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:13.535714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:13.535674    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:14.422861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:14.422825    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:15.251792 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:15.251754    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:15.422228 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:15.422186    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:16.613310 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:16.609618    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:16.801120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:16.801029    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:17.000767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:17.000727    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:17.441216 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:17.440850    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:17.810601 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:17.810566    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:18.790992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:18.790951    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:18.828302 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:39:18.828257    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:39:19.875906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:19.875580    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:20.119035 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:20.119000    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:20.863960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:20.863905    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:21.710894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:21.710488    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:22.537488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:22.537449    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:23.044727 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:23.044693    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"]
Oct 31 13:39:23.049884 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:23.049834    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-636e4838-sdprf"]
Oct 31 13:39:23.336432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:23.336338    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:24.311123 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:24.311091    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:24.782266 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:24.782217    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad597eab-3ce1-44a8-8e37-cd64c3c83657" path="/var/lib/kubelet/pods/ad597eab-3ce1-44a8-8e37-cd64c3c83657/volumes"
Oct 31 13:39:25.789064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:25.789027    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:26.197949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:26.190471    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:27.330832 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:27.330794    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:28.208442 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:28.208399    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:29.397120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:29.397082    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:29.662276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:29.662151    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:31.037375 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:31.037332    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:32.161668 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:32.161543    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:32.996221 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:32.996136    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:33.022099 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:33.022062    4428 scope.go:122] "RemoveContainer" containerID="20cf6d9bcdd41646e1967fe5e6a849a7feb37d5400f84c66b62d9939333be5ef"
Oct 31 13:39:33.909938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:33.909790    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:34.725079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:34.725043    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:34.984951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:34.983925    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:35.914570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:35.914536    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:36.852573 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:36.852532    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:37.325136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:37.325106    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:37.932511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:37.932465    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:38.763626 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:38.763589    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:39.792555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:39.792407    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:40.200824 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:40.200723    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:40.960707 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:40.960300    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:41.038074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:41.038031    4428 patch_prober.go:29] interesting pod/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:39:41.038294 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:41.038116    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf" podUID="2ccec53f-f517-420f-b20d-2604a73f2884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:39:41.592831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:41.592800    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:42.278614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:42.278498    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:43.210497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:43.205881    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:43.614202 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:43.613034    4428 patch_prober.go:29] interesting pod/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": read tcp 10.134.0.2:54124->10.134.0.190:15021: read: connection reset by peer" start-of-body=
Oct 31 13:39:43.614202 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:43.613111    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": read tcp 10.134.0.2:54124->10.134.0.190:15021: read: connection reset by peer"
Oct 31 13:39:43.916983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:43.909705    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:44.272487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:44.272397    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:44.671032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:44.670832    4428 generic.go:356] "Generic (PLEG): container finished" podID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerID="9fb715f96f38b48403dcf2b346071307d3e228d24b0cac0cc53101e386d544c5" exitCode=0
Oct 31 13:39:44.671634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:44.671091    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" event={"ID":"d1af67f4-a0eb-402b-8453-7cb5ab463ab1","Type":"ContainerDied","Data":"9fb715f96f38b48403dcf2b346071307d3e228d24b0cac0cc53101e386d544c5"}
Oct 31 13:39:44.672117 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:44.672079    4428 scope.go:122] "RemoveContainer" containerID="9fb715f96f38b48403dcf2b346071307d3e228d24b0cac0cc53101e386d544c5"
Oct 31 13:39:44.677934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:44.677900    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf" event={"ID":"2ccec53f-f517-420f-b20d-2604a73f2884","Type":"ContainerDied","Data":"fddf8f8bc7bb0e249ae0e4d6dbd84845abd85605e4ca349ffb8d374bba279f5a"}
Oct 31 13:39:44.678102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:44.677927    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:39:44.678102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:44.678026    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:39:44.678102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:44.678075    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:39:44.678280 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:44.677852    4428 generic.go:356] "Generic (PLEG): container finished" podID="2ccec53f-f517-420f-b20d-2604a73f2884" containerID="fddf8f8bc7bb0e249ae0e4d6dbd84845abd85605e4ca349ffb8d374bba279f5a" exitCode=0
Oct 31 13:39:44.678488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:44.678472    4428 scope.go:122] "RemoveContainer" containerID="fddf8f8bc7bb0e249ae0e4d6dbd84845abd85605e4ca349ffb8d374bba279f5a"
Oct 31 13:39:44.682581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:44.682529    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:39:44.682724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:44.682624    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:39:44.682724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:44.682659    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:39:45.689097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:45.686414    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:45.776265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:45.776222    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" event={"ID":"d1af67f4-a0eb-402b-8453-7cb5ab463ab1","Type":"ContainerStarted","Data":"568dc5b3f98f4a9f69d2e9e48db34228213d897cf4eac287edc5071280cc1340"}
Oct 31 13:39:45.782785 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:45.782750    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf" event={"ID":"2ccec53f-f517-420f-b20d-2604a73f2884","Type":"ContainerStarted","Data":"d9f56ee82b3c5380e7564ee713f0b9b1f5a0beb7a13cec339c1bb030a1449575"}
Oct 31 13:39:45.942190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:45.941729    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:39:45.942190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:45.941853    4428 patch_prober.go:29] interesting pod/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 13:39:45.942190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:45.941957    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 13:39:45.942837 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:45.942809    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:46.480764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:46.480729    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:39:46.489695 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:46.484038    4428 patch_prober.go:29] interesting pod/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 31 13:39:46.489695 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:46.484094    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf" podUID="2ccec53f-f517-420f-b20d-2604a73f2884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 31 13:39:46.724350 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:46.724311    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:46.940977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:46.940852    4428 patch_prober.go:29] interesting pod/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 13:39:46.941273 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:46.941241    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 13:39:47.038735 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:47.038703    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"]
Oct 31 13:39:47.044684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:47.044651    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da34bff6-h7p22"]
Oct 31 13:39:47.481706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:47.481255    4428 patch_prober.go:29] interesting pod/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 31 13:39:47.481706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:47.481316    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf" podUID="2ccec53f-f517-420f-b20d-2604a73f2884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 31 13:39:47.950769 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:47.950666    4428 patch_prober.go:29] interesting pod/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 13:39:47.955111 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:47.955049    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 13:39:48.110536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:48.110491    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:48.484384 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:48.484336    4428 patch_prober.go:29] interesting pod/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 31 13:39:48.484729 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:48.484695    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf" podUID="2ccec53f-f517-420f-b20d-2604a73f2884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 31 13:39:48.773367 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:39:48.773263    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:39:48.803824 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:48.803779    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08fd0053-54bb-4fb3-85c5-50ef5c94e2ff" path="/var/lib/kubelet/pods/08fd0053-54bb-4fb3-85c5-50ef5c94e2ff/volumes"
Oct 31 13:39:48.958400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:48.958359    4428 patch_prober.go:29] interesting pod/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 13:39:48.959144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:48.959109    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 13:39:49.068571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:49.068473    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:49.481356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:49.481313    4428 patch_prober.go:29] interesting pod/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 31 13:39:49.481688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:49.481656    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf" podUID="2ccec53f-f517-420f-b20d-2604a73f2884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 31 13:39:49.933963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:49.931556    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:49.960904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:49.950496    4428 patch_prober.go:29] interesting pod/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 13:39:49.960904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:49.950634    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 13:39:50.481893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:50.481301    4428 patch_prober.go:29] interesting pod/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 31 13:39:50.481893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:50.481388    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf" podUID="2ccec53f-f517-420f-b20d-2604a73f2884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 31 13:39:50.706200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:50.706120    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:50.943437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:50.943342    4428 patch_prober.go:29] interesting pod/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 13:39:50.943767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:50.943731    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 13:39:51.486676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:51.483236    4428 patch_prober.go:29] interesting pod/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 31 13:39:51.486676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:51.483300    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf" podUID="2ccec53f-f517-420f-b20d-2604a73f2884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 31 13:39:51.711907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:51.710449    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:51.941440 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:51.941329    4428 patch_prober.go:29] interesting pod/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 13:39:51.941440 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:51.941400    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 13:39:52.433505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:52.433475    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:52.481451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:52.481404    4428 patch_prober.go:29] interesting pod/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 31 13:39:52.481790 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:52.481743    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf" podUID="2ccec53f-f517-420f-b20d-2604a73f2884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 31 13:39:52.615683 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:52.615596    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:52.902088 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:52.902053    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:52.941275 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:52.941233    4428 patch_prober.go:29] interesting pod/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 13:39:52.941479 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:52.941305    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 13:39:53.507353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:53.507315    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:53.727732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:53.727676    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:53.869284 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:53.869199    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:39:53.871329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:53.871305    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:39:53.910893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:53.906278    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:39:53.983343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:53.983311    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:39:53.985443 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:53.985420    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:39:54.000213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:54.000181    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:39:55.207846 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:55.207775    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:56.040562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:56.040488    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:56.572982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:56.572944    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:57.003961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:56.973794    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:57.935482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:57.935446    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:58.971839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:58.971802    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:39:59.803551 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:39:59.803511    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:00.859817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:00.859754    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:01.908453 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:01.908414    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:02.750988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:02.746384    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:03.650729 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:03.650693    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:04.819951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:04.818477    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:06.267894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:06.263011    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:06.421985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:06.421942    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:06.510682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:06.509626    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:40:06.738448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:06.737393    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:06.738448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:06.737986    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:40:07.225544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:07.223988    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:07.362555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:07.361099    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 13:40:07.455891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:07.452322    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:40:07.456894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:07.456352    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:40:07.477979 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:07.477856    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:07.517201 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:07.517155    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:07.518252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:07.518226    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:40:07.655034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:07.654987    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:07.785952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:07.758967    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 13:40:07.785952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:07.759429    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 13:40:07.785952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:07.760039    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:40:07.785952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:07.760095    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:40:08.178842 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:08.178720    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:08.235904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:08.229042    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:08.235904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:08.230066    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:08.246895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:08.238664    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:40:08.281450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:08.281405    4428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded"
Oct 31 13:40:08.376916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:08.376431    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:08.440945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:08.436244    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:40:08.469897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:08.466983    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:08.637892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:08.573960    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:08.762208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:08.630116    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 13:40:08.762208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:08.761137    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:08.762208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:08.722204    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:40:09.111957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:09.098018    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 13:40:09.111957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:09.098099    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:40:09.111957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:09.098136    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 13:40:09.111957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:09.098168    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 13:40:09.111957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:09.098203    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 13:40:09.113544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:09.113288    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:10.376075 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:10.376017    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:11.568752 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:11.568697    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:12.879893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:12.872539    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:13.602738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:13.602689    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:13.767547 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:13.767515    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:13.935950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:13.935796    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:15.061926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:15.061780    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:15.938967 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:15.936464    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:16.827892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:16.824101    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:17.633118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:17.632536    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:18.517829 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:18.517790    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:18.747219 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:40:18.747172    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:40:19.485559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:19.485516    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:19.795271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:19.667009    4428 patch_prober.go:29] interesting pod/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:40:19.795271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:19.667073    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp" podUID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:40:20.219131 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:20.219077    4428 patch_prober.go:29] interesting pod/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": read tcp 10.134.0.2:51158->10.134.0.194:15021: read: connection reset by peer" start-of-body=
Oct 31 13:40:20.219383 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:20.219143    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx" podUID="6349679a-67f7-41e5-b289-dc91302b5977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": read tcp 10.134.0.2:51158->10.134.0.194:15021: read: connection reset by peer"
Oct 31 13:40:20.332899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:20.327460    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:20.336728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:20.336688    4428 generic.go:356] "Generic (PLEG): container finished" podID="6349679a-67f7-41e5-b289-dc91302b5977" containerID="ef28b27b58d8d70c68447f988100a8adc7d58ec0c399ab24db584a2f8a845c7f" exitCode=0
Oct 31 13:40:20.336988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:20.336931    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx" event={"ID":"6349679a-67f7-41e5-b289-dc91302b5977","Type":"ContainerDied","Data":"ef28b27b58d8d70c68447f988100a8adc7d58ec0c399ab24db584a2f8a845c7f"}
Oct 31 13:40:20.337613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:20.337526    4428 scope.go:122] "RemoveContainer" containerID="ef28b27b58d8d70c68447f988100a8adc7d58ec0c399ab24db584a2f8a845c7f"
Oct 31 13:40:20.340318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:20.340281    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:40:20.340452 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:20.340381    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:40:20.340452 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:20.340433    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:40:20.348758 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:20.348711    4428 generic.go:356] "Generic (PLEG): container finished" podID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" containerID="2a73b302ef8ff6424a483a0adfe6e5967a7ec5a467db474c3b2bcd12b1ad5517" exitCode=0
Oct 31 13:40:20.349468 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:20.349441    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp" event={"ID":"ace3362d-9bb7-46e2-b218-4e7d3e0ed354","Type":"ContainerDied","Data":"2a73b302ef8ff6424a483a0adfe6e5967a7ec5a467db474c3b2bcd12b1ad5517"}
Oct 31 13:40:20.350500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:20.350480    4428 scope.go:122] "RemoveContainer" containerID="2a73b302ef8ff6424a483a0adfe6e5967a7ec5a467db474c3b2bcd12b1ad5517"
Oct 31 13:40:20.370941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:20.370568    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:40:20.370941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:20.370687    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:40:20.370941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:20.370748    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:40:21.465894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:21.463128    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx" event={"ID":"6349679a-67f7-41e5-b289-dc91302b5977","Type":"ContainerStarted","Data":"becc1a45e7795362df9130b5c399494a36ab0e5ced8c46a99b58f9e53257b719"}
Oct 31 13:40:21.669379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:21.660223    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp" event={"ID":"ace3362d-9bb7-46e2-b218-4e7d3e0ed354","Type":"ContainerStarted","Data":"4ad106c9ce07fd90a1994a884773eb13ec19a77c5fa44fb8675dd543be7c005e"}
Oct 31 13:40:21.765963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:21.759149    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:22.018700 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:22.018597    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:40:22.018925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:22.018740    4428 patch_prober.go:29] interesting pod/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 13:40:22.018925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:22.018791    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx" podUID="6349679a-67f7-41e5-b289-dc91302b5977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 13:40:22.598011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:22.597298    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:40:22.598011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:22.597414    4428 patch_prober.go:29] interesting pod/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 13:40:22.598011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:22.597474    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp" podUID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 13:40:22.951516 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:22.951418    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:23.028544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:23.027860    4428 patch_prober.go:29] interesting pod/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 13:40:23.028839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:23.028809    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx" podUID="6349679a-67f7-41e5-b289-dc91302b5977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 13:40:23.593900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:23.592476    4428 patch_prober.go:29] interesting pod/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 13:40:23.593900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:23.592573    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp" podUID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 13:40:23.847302 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:23.843902    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:24.020341 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:24.020301    4428 patch_prober.go:29] interesting pod/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 13:40:24.020655 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:24.020620    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx" podUID="6349679a-67f7-41e5-b289-dc91302b5977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 13:40:24.531901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:24.529961    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:24.603896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:24.597493    4428 patch_prober.go:29] interesting pod/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 13:40:24.603896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:24.597554    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp" podUID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 13:40:24.838555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:24.838446    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:25.041900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:25.031176    4428 patch_prober.go:29] interesting pod/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 13:40:25.099080 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:25.092191    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx" podUID="6349679a-67f7-41e5-b289-dc91302b5977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 13:40:25.099080 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:25.072126    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:25.502484 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:25.502441    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:25.597378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:25.597340    4428 patch_prober.go:29] interesting pod/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 13:40:25.597705 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:25.597661    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp" podUID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 13:40:25.971906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:25.970421    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:26.029684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:26.018797    4428 patch_prober.go:29] interesting pod/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 13:40:26.030078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:26.030040    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx" podUID="6349679a-67f7-41e5-b289-dc91302b5977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 13:40:26.258614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:26.258526    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:26.591959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:26.591141    4428 patch_prober.go:29] interesting pod/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 13:40:26.591959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:26.591211    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp" podUID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 13:40:27.019524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:27.019488    4428 patch_prober.go:29] interesting pod/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 13:40:27.019946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:27.019862    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx" podUID="6349679a-67f7-41e5-b289-dc91302b5977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 13:40:27.349639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:27.349544    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:27.588888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:27.588832    4428 patch_prober.go:29] interesting pod/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 13:40:27.589203 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:27.589173    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp" podUID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 13:40:28.019023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:28.018989    4428 patch_prober.go:29] interesting pod/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 13:40:28.019252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:28.019057    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx" podUID="6349679a-67f7-41e5-b289-dc91302b5977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 13:40:28.304916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:28.304780    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:28.636803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:28.636712    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:40:28.638281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:28.638251    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:40:28.644462 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:28.644431    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:40:29.024643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:29.024612    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:40:29.024919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:29.024899    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:40:29.025795 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:29.025774    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:40:29.217897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:29.217554    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:30.193077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:30.169267    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:31.135902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:31.134224    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:32.090007 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:32.089973    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:33.126057 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:33.126023    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:33.312546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:33.312509    4428 scope.go:122] "RemoveContainer" containerID="adb3e667aa9378dae670da1c4ce7fa9e31370ba4aeaad5d3dea9d58341b710dd"
Oct 31 13:40:33.323944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:33.323881    4428 scope.go:122] "RemoveContainer" containerID="66c30904050b44452fd6c759f13ba29063c5cfa15dce940448579aadb0df6320"
Oct 31 13:40:33.334603 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:33.334576    4428 scope.go:122] "RemoveContainer" containerID="1e2b378710a40083e48f83f8dcdaa4ded3a77ab8c6fc5ffa5d5ef77fff2e7a0c"
Oct 31 13:40:34.014895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:34.014425    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:34.947911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:34.946019    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:35.977778 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:35.977738    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:36.814952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:36.814348    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:37.620628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:37.620586    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:38.517894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:38.508894    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:39.303827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:39.303785    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:40.109939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:40.099367    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:41.017105 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:41.006144    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:42.049456 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:42.049416    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:42.899726 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:42.899603    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:43.815013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:43.814976    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:44.717191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:44.717111    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:45.648959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:45.648844    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:46.511896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:46.511026    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:47.466592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:47.466553    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:48.313972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:48.313930    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:48.742745 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:40:48.742704    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:40:49.238907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:49.238680    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:50.015899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:50.011539    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:50.892022 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:50.891984    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:51.951750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:51.951014    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:52.950208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:52.950173    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:54.093989 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:54.093951    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:54.847056 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:54.847015    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:55.080268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:55.080237    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:55.328893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:55.325169    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:56.151333 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:56.151301    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:56.371936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:56.353531    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:56.879251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:56.879222    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:58.032766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:58.021172    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:58.994670 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:58.994623    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:40:59.896618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:40:59.896580    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:00.692635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:00.692579    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:01.633662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:01.633596    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:02.432744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:02.430120    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:03.304549 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:03.304450    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:04.108990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:04.108913    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"]
Oct 31 13:41:04.118579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:04.118511    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41939f60-hbqs4"]
Oct 31 13:41:04.121602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:04.121552    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:04.758860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:04.758826    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c92c3cd-9053-4a35-925c-939e82694732" path="/var/lib/kubelet/pods/9c92c3cd-9053-4a35-925c-939e82694732/volumes"
Oct 31 13:41:05.232892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:05.227963    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:06.505249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:06.505215    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:07.640917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:07.640854    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:08.665715 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:08.665678    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:09.954426 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:09.954322    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:10.926755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:10.926720    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:11.857019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:11.856984    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:12.726918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:12.725531    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:13.501897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:13.494253    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:13.706693 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:13.706661    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:14.355655 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:14.355586    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:15.106890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:15.106819    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:15.891899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:15.891704    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:16.833732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:16.833618    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:17.694353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:17.694288    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:18.528421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:18.528383    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:18.745132 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:41:18.745086    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:41:19.518766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:19.518725    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:19.727465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:19.727427    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:20.706898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:20.705565    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:21.459894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:21.458483    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:21.640815 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:21.640787    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:21.817463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:21.817368    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:22.204210 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:22.204108    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:23.355069 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:23.355010    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:24.036893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:24.034563    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:24.820155 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:24.816830    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:25.710194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:25.710152    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:26.601588 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:26.601547    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:27.408024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:27.407837    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:28.293708 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:28.293671    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:29.226957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:29.226918    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:30.034920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:30.033841    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:30.976611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:30.976449    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:31.851824 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:31.851786    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:32.628893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:32.626319    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:33.379820 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:33.379783    4428 scope.go:122] "RemoveContainer" containerID="f7f0ba019e1e63f5fc74f651df5ffcac74fa2ffabfd4502be23eacb5b775253e"
Oct 31 13:41:33.395091 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:33.395040    4428 scope.go:122] "RemoveContainer" containerID="6cb1a2fc3a59ff0cfafdbdcb2b4435a01b4e1fb76170219bd82c3a0d63e0349e"
Oct 31 13:41:33.411159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:33.411136    4428 scope.go:122] "RemoveContainer" containerID="984ea124b5d39fcd2e0dbb76bf71260ae041f1b7ea3e399d86d8935b98881cae"
Oct 31 13:41:33.530915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:33.530823    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:34.453574 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:34.447716    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:34.960165 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:34.960098    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:35.380290 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:35.380135    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:36.331788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:36.315913    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:37.112694 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:37.112552    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:37.932799 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:37.924305    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:38.841813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:38.841772    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:39.008567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:39.008537    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:39.916001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:39.915960    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:40.708166 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:40.708111    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:41.644986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:41.644928    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:42.508896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:42.508739    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:43.346899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:43.343441    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:44.208190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:44.208149    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:45.131312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:45.131270    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:45.854555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:45.854464    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:46.827045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:46.826937    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:47.721741 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:47.721697    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:48.452962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:48.452926    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:48.744724 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:41:48.744678    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:41:49.317185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:49.311397    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:50.168125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:50.166712    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:50.954896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:50.953570    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:51.725895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:51.722512    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:52.644890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:52.644826    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:53.677895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:53.676291    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:54.771665 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:54.771625    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:55.432494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:55.432458    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:56.341160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:56.341122    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:57.126896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:57.124530    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:57.892949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:57.892912    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:58.702753 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:58.702700    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:41:59.517891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:41:59.511838    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:00.318638 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:00.318574    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:01.190016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:01.189978    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:02.014839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:02.014796    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:02.721955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:02.721916    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:03.684893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:03.684429    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:04.521431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:04.521393    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:05.273556 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:05.273496    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:05.594748 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:05.594643    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:06.325895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:06.325834    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:07.492732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:07.492692    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:08.469686 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:08.469648    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:09.238892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:09.238825    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:10.197072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:10.197000    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:11.008532 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:11.008480    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:11.354969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:11.354825    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:11.808147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:11.808115    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:12.913175 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:12.913133    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:14.086891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:14.063858    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:14.336482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:14.336440    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:14.561922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:14.561841    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:14.926576 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:14.926446    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:15.898892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:15.895557    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:16.048834 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:16.048790    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"]
Oct 31 13:42:16.054098 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:16.054068    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aa93a88-h8zrs"]
Oct 31 13:42:16.760205 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:16.760169    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79bf625f-03f6-4d45-8227-683270c01391" path="/var/lib/kubelet/pods/79bf625f-03f6-4d45-8227-683270c01391/volumes"
Oct 31 13:42:17.062964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:17.062803    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:18.143137 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:18.120836    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:18.742987 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:42:18.742953    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:42:19.080942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:19.080678    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:20.060836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:20.060800    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:20.901819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:20.895340    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:21.909468 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:21.909429    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:22.723939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:22.723885    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:23.513366 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:23.513325    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:24.502271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:24.502222    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:25.092201 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:25.092167    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:25.259405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:25.259351    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:26.100187 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:26.100149    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:27.138197 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:27.138168    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:27.311203 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:27.311156    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:28.317759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:28.317718    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:29.143926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:29.143886    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:30.135896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:30.130183    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:31.070330 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:31.070277    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:31.926470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:31.926403    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:32.755391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:32.755334    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:32.921328 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:32.921281    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:33.456491 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:33.456457    4428 scope.go:122] "RemoveContainer" containerID="293da6842dade8d063f319ae06d6be691257d1d8f52e89184d3070256363797c"
Oct 31 13:42:33.468748 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:33.468722    4428 scope.go:122] "RemoveContainer" containerID="00793ed6d6235d8a605668c66fc76e94b1e39d6cd20c75a6ed1c7dc860aed721"
Oct 31 13:42:33.480955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:33.480932    4428 scope.go:122] "RemoveContainer" containerID="ebce4d901a51e43baa4261935975c8b59f6133b9c466ea47f91d03138ee7f36d"
Oct 31 13:42:33.492123 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:33.492100    4428 scope.go:122] "RemoveContainer" containerID="748a3e5a407aab8dfe2a69ab9c201372f648af409538b86ab96a639c3410ec6c"
Oct 31 13:42:33.502529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:33.502507    4428 scope.go:122] "RemoveContainer" containerID="5f4aa9fa07437bbb943eb2a2762a625bc29e635735b6ee20dbf03405950d7794"
Oct 31 13:42:34.035554 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:34.035506    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:34.917941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:34.915408    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:35.668231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:35.668198    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:35.833835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:35.832003    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:36.517916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:36.517857    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:37.398543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:37.398505    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:38.128890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:38.120847    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:39.016737 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:39.016590    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:39.811665 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:39.811626    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:40.622335 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:40.622255    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:41.548707 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:41.548659    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:42.372697 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:42.372662    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:43.118696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:43.118658    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:44.104694 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:44.103689    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:44.663264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:44.663231    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:44.827291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:44.827238    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:44.998643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:44.998587    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:45.594057 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:45.594020    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:46.518700 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:46.518666    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:47.334895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:47.325501    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:48.193435 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:48.193399    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:48.743895 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:42:48.743841    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:42:49.107661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:49.107560    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:49.989890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:49.986876    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:50.833090 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:50.833057    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:52.088037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:52.087982    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:53.357900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:53.354858    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:56.057888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:56.052483    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:57.311307 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:57.311206    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:58.255895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:58.247677    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:42:59.317893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:42:59.310684    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:00.169291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:00.169252    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:00.471404 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:00.471352    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:00.656093 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:00.656065    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:00.996249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:00.996184    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:01.831490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:01.831445    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:02.693963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:02.693922    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:03.426123 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:03.426087    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:04.291176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:04.291078    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:05.117539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:05.116561    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:05.824713 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:05.824675    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:06.656900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:06.639437    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:07.325748 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:07.325710    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:08.038916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:08.035577    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:09.028527 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:09.028484    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:09.775545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:09.775510    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:10.513910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:10.509793    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:11.428706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:11.419700    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:12.208889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:12.208554    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:12.972116 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:12.972074    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:13.905896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:13.905525    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:14.090732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:14.090694    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:14.542394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:14.542360    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:14.821952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:14.813421    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:15.629898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:15.629843    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:16.412585 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:16.412545    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:17.263835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:17.263788    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:18.005106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:18.005069    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:18.743505 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:43:18.743468    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:43:18.798968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:18.794854    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:19.661245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:19.659105    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:20.429367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:20.429332    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:21.326044 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:21.325992    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:22.137694 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:22.134686    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:23.147605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:23.147568    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:23.741214 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:23.741176    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:24.627735 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:24.627685    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:25.507895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:25.507053    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:26.297894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:26.292465    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:27.133851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:27.133807    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:27.953446 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:27.953406    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:28.827586 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:28.827452    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:29.720926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:29.720846    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:30.512062 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:30.512009    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:31.485105 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:31.485062    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:31.682090 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:31.682053    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:32.621490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:32.621442    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:32.798084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:32.798051    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:33.564385 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:33.564356    4428 scope.go:122] "RemoveContainer" containerID="1cdb4eb5d2d9f9a9b579c1078ce8537a4bed16697b50b1a117b26c07cbb39727"
Oct 31 13:43:33.576134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:33.576105    4428 scope.go:122] "RemoveContainer" containerID="d2a0d9dfd6a662705da026c249538fc4f0e8a91bf309cd42d23866fbef9348fd"
Oct 31 13:43:33.734572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:33.731284    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:34.252554 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:34.252517    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:34.432585 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:34.432553    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:34.676233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:34.676146    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:35.658902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:35.658842    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:36.685150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:36.685105    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:37.529366 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:37.529328    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:38.628894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:38.627996    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:39.609637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:39.609600    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:40.642254 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:40.642218    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:41.678931 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:41.678890    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:42.606899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:42.597033    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:43.747626 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:43.747580    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:45.268964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:45.268922    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:46.376984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:46.376951    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:47.607524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:47.607471    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:48.723549 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:48.723508    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:48.818248 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:43:48.818211    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:43:49.048854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:49.048756    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:50.305099 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:50.305063    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:51.375894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:51.364323    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:52.280893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:52.280443    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:53.347637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:53.347597    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:54.415138 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:54.415104    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:55.475764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:55.475705    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:56.331425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:56.331382    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:56.469288 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:56.469250    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"]
Oct 31 13:43:56.469553 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:56.469527    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7" podUID="0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" containerName="istio-proxy" containerID="cri-o://21dd50dc671e562caf98d2302522f322858d85c127b02fb507b6a36edaec3fc7" gracePeriod=30
Oct 31 13:43:57.245915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:57.245858    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"]
Oct 31 13:43:57.246391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:57.246088    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt" podUID="50a90471-68bd-4059-ba8b-8719805293e8" containerName="istio-proxy" containerID="cri-o://5aa2ef4d9cfa75cdacb5ebe5f9137144e0c359afdf8a1073e0982001b94f07b5" gracePeriod=30
Oct 31 13:43:57.542300 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:57.542167    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:58.543602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:58.543567    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:59.376765 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:59.373946    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"]
Oct 31 13:43:59.383227 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:59.383192    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"]
Oct 31 13:43:59.587480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:59.587395    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:43:59.781448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:43:59.781388    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:00.721081 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:00.721004    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:01.186389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.186309    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"]
Oct 31 13:44:01.186566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.186543    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6" podUID="37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" containerName="istio-proxy" containerID="cri-o://00f748be89a81f9ff06efc0c3f850515ca8fe06d72b33d04e931819ae67f8429" gracePeriod=30
Oct 31 13:44:01.773428 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.773355    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:44:01.800936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.800898    4428 generic.go:356] "Generic (PLEG): container finished" podID="0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" containerID="21dd50dc671e562caf98d2302522f322858d85c127b02fb507b6a36edaec3fc7" exitCode=0
Oct 31 13:44:01.801144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.801004    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"
Oct 31 13:44:01.801144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.801037    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7" event={"ID":"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0","Type":"ContainerDied","Data":"21dd50dc671e562caf98d2302522f322858d85c127b02fb507b6a36edaec3fc7"}
Oct 31 13:44:01.801144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.801106    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7" event={"ID":"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0","Type":"ContainerDied","Data":"7763b10f1b8e8261dd743887ca8555ae80a04816459f178e5ca539a7f3637416"}
Oct 31 13:44:01.801144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.801138    4428 scope.go:122] "RemoveContainer" containerID="21dd50dc671e562caf98d2302522f322858d85c127b02fb507b6a36edaec3fc7"
Oct 31 13:44:01.882494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.880411    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-credential-socket\") pod \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") "
Oct 31 13:44:01.882494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.880493    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-podinfo\") pod \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") "
Oct 31 13:44:01.882494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.880567    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-workload-socket\") pod \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") "
Oct 31 13:44:01.882494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.880774    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-workload-certs\") pod \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") "
Oct 31 13:44:01.882494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.880841    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-envoy\") pod \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") "
Oct 31 13:44:01.882494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.880907    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-kube-api-access-zlqrj\" (UniqueName: \"kubernetes.io/projected/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-kube-api-access-zlqrj\") pod \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") "
Oct 31 13:44:01.882494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.880967    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-data\") pod \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") "
Oct 31 13:44:01.882494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.881066    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-token\" (UniqueName: \"kubernetes.io/projected/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-token\") pod \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") "
Oct 31 13:44:01.882494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.881099    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istiod-ca-cert\") pod \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\" (UID: \"0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0\") "
Oct 31 13:44:01.882494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.881464    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-workload-certs" pod "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" (UID: "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:01.882494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.881703    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-credential-socket" pod "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" (UID: "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:01.882494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.882150    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:01.882494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.882298    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:01.885628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.885417    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-envoy" pod "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" (UID: "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:01.885628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.885577    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:01.885628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.885577    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-data" pod "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" (UID: "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:01.886028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.885679    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istiod-ca-cert" pod "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" (UID: "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:01.886329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.886302    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-workload-socket" pod "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" (UID: "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:01.887595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.887569    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-podinfo" pod "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" (UID: "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:44:01.888335 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.888311    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-kube-api-access-zlqrj" pod "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" (UID: "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0"). InnerVolumeSpecName "kube-api-access-zlqrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:01.891115 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.891089    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"]
Oct 31 13:44:01.892045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.892007    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-token" pod "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" (UID: "0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:01.892311 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.892294    4428 cpu_manager.go:414] "containerMap: removing container" podUID="0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" containerName="istio-proxy"
Oct 31 13:44:01.892372 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.892313    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" containerName="istio-proxy"
Oct 31 13:44:01.892484 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.892472    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" containerName="istio-proxy"
Oct 31 13:44:01.908293 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.897731    4428 scope.go:122] "RemoveContainer" containerID="21dd50dc671e562caf98d2302522f322858d85c127b02fb507b6a36edaec3fc7"
Oct 31 13:44:01.909573 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:44:01.909524    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21dd50dc671e562caf98d2302522f322858d85c127b02fb507b6a36edaec3fc7\": container with ID starting with 21dd50dc671e562caf98d2302522f322858d85c127b02fb507b6a36edaec3fc7 not found: ID does not exist" containerID="21dd50dc671e562caf98d2302522f322858d85c127b02fb507b6a36edaec3fc7"
Oct 31 13:44:01.909787 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.909573    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21dd50dc671e562caf98d2302522f322858d85c127b02fb507b6a36edaec3fc7"} err="failed to get container status \"21dd50dc671e562caf98d2302522f322858d85c127b02fb507b6a36edaec3fc7\": rpc error: code = NotFound desc = could not find container \"21dd50dc671e562caf98d2302522f322858d85c127b02fb507b6a36edaec3fc7\": container with ID starting with 21dd50dc671e562caf98d2302522f322858d85c127b02fb507b6a36edaec3fc7 not found: ID does not exist"
Oct 31 13:44:01.945348 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:01.942459    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.033928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.032958    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"]
Oct 31 13:44:02.035540 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.035504    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-data\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.035704 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.035557    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq6g4\" (UniqueName: \"kubernetes.io/projected/db1e5779-e42f-48ed-9804-31af5494ce8a-kube-api-access-cq6g4\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.035704 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.035613    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-credential-socket\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.035704 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.035685    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-podinfo\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.035990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.035714    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-token\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.035990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.035808    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-envoy\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.035990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.035909    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1e5779-e42f-48ed-9804-31af5494ce8a-istiod-ca-cert\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.035990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.035939    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-workload-certs\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.036262 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.036001    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-workload-socket\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.036347 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.036327    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:02.036405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.036353    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:02.036405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.036366    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:02.036405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.036380    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:02.036405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.036394    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlqrj\" (UniqueName: \"kubernetes.io/projected/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-kube-api-access-zlqrj\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:02.036627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.036408    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:02.036627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.036422    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:02.136185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.136137    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"]
Oct 31 13:44:02.138590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.138549    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-workload-socket\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.138999 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.138862    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-data\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.138999 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.138918    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq6g4\" (UniqueName: \"kubernetes.io/projected/db1e5779-e42f-48ed-9804-31af5494ce8a-kube-api-access-cq6g4\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.138999 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.138972    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-credential-socket\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.139298 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.139041    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-podinfo\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.139298 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.139072    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-token\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.139298 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.139165    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-envoy\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.139298 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.139173    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-workload-socket\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.139298 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.139230    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-data\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.139298 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.139263    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1e5779-e42f-48ed-9804-31af5494ce8a-istiod-ca-cert\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.139298 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.139295    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-workload-certs\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.139698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.139611    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-workload-certs\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.139889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.139840    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-credential-socket\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.140042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.140011    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1e5779-e42f-48ed-9804-31af5494ce8a-istiod-ca-cert\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.141806 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.141782    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-envoy\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.142265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.142246    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-podinfo\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.145172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.145149    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vcml7"]
Oct 31 13:44:02.153235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.153213    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-token\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.153344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.153329    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq6g4\" (UniqueName: \"kubernetes.io/projected/db1e5779-e42f-48ed-9804-31af5494ce8a-kube-api-access-cq6g4\") pod \"e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") " pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.347188 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.347095    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:02.633560 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.633530    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:44:02.714805 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:44:02.714777    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1e5779_e42f_48ed_9804_31af5494ce8a.slice/crio-a692a5dc099d4126d0fbe8387c8c3eaf6e865c1270522cb0b5fbf2b25256f7c5 WatchSource:0}: Error finding container a692a5dc099d4126d0fbe8387c8c3eaf6e865c1270522cb0b5fbf2b25256f7c5: Status 404 returned error can't find the container with id a692a5dc099d4126d0fbe8387c8c3eaf6e865c1270522cb0b5fbf2b25256f7c5
Oct 31 13:44:02.717706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.717674    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"]
Oct 31 13:44:02.717861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.717830    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:44:02.717960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.717944    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:44:02.717998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.717982    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:44:02.751603 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.751575    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a90471-68bd-4059-ba8b-8719805293e8-kube-api-access-f9czv\" (UniqueName: \"kubernetes.io/projected/50a90471-68bd-4059-ba8b-8719805293e8-kube-api-access-f9czv\") pod \"50a90471-68bd-4059-ba8b-8719805293e8\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") "
Oct 31 13:44:02.751791 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.751666    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-workload-socket\") pod \"50a90471-68bd-4059-ba8b-8719805293e8\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") "
Oct 31 13:44:02.751952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.751813    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50a90471-68bd-4059-ba8b-8719805293e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a90471-68bd-4059-ba8b-8719805293e8-istio-podinfo\") pod \"50a90471-68bd-4059-ba8b-8719805293e8\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") "
Oct 31 13:44:02.751952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.751940    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-workload-socket" pod "50a90471-68bd-4059-ba8b-8719805293e8" (UID: "50a90471-68bd-4059-ba8b-8719805293e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:02.752096 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.752030    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-workload-certs\") pod \"50a90471-68bd-4059-ba8b-8719805293e8\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") "
Oct 31 13:44:02.752155 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.752106    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-credential-socket\") pod \"50a90471-68bd-4059-ba8b-8719805293e8\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") "
Oct 31 13:44:02.752207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.752178    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-istio-data\") pod \"50a90471-68bd-4059-ba8b-8719805293e8\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") "
Oct 31 13:44:02.760028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.752390    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a90471-68bd-4059-ba8b-8719805293e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a90471-68bd-4059-ba8b-8719805293e8-istiod-ca-cert\") pod \"50a90471-68bd-4059-ba8b-8719805293e8\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") "
Oct 31 13:44:02.760028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.752468    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-istio-envoy\") pod \"50a90471-68bd-4059-ba8b-8719805293e8\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") "
Oct 31 13:44:02.760028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.752501    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a90471-68bd-4059-ba8b-8719805293e8-istio-token\" (UniqueName: \"kubernetes.io/projected/50a90471-68bd-4059-ba8b-8719805293e8-istio-token\") pod \"50a90471-68bd-4059-ba8b-8719805293e8\" (UID: \"50a90471-68bd-4059-ba8b-8719805293e8\") "
Oct 31 13:44:02.760028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.753465    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:02.760028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.755133    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a90471-68bd-4059-ba8b-8719805293e8-kube-api-access-f9czv" pod "50a90471-68bd-4059-ba8b-8719805293e8" (UID: "50a90471-68bd-4059-ba8b-8719805293e8"). InnerVolumeSpecName "kube-api-access-f9czv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:02.760028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.755365    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-credential-socket" pod "50a90471-68bd-4059-ba8b-8719805293e8" (UID: "50a90471-68bd-4059-ba8b-8719805293e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:02.760028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.755995    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-istio-data" pod "50a90471-68bd-4059-ba8b-8719805293e8" (UID: "50a90471-68bd-4059-ba8b-8719805293e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:02.760028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.756326    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a90471-68bd-4059-ba8b-8719805293e8-istiod-ca-cert" pod "50a90471-68bd-4059-ba8b-8719805293e8" (UID: "50a90471-68bd-4059-ba8b-8719805293e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:02.760028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.756373    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-workload-certs" pod "50a90471-68bd-4059-ba8b-8719805293e8" (UID: "50a90471-68bd-4059-ba8b-8719805293e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:02.760028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.758033    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50a90471-68bd-4059-ba8b-8719805293e8-istio-podinfo" pod "50a90471-68bd-4059-ba8b-8719805293e8" (UID: "50a90471-68bd-4059-ba8b-8719805293e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:44:02.760028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.758970    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-istio-envoy" pod "50a90471-68bd-4059-ba8b-8719805293e8" (UID: "50a90471-68bd-4059-ba8b-8719805293e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:02.765550 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.765523    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a90471-68bd-4059-ba8b-8719805293e8-istio-token" pod "50a90471-68bd-4059-ba8b-8719805293e8" (UID: "50a90471-68bd-4059-ba8b-8719805293e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:02.768206 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.768178    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0" path="/var/lib/kubelet/pods/0ecf6cba-0a6c-40b4-a7f1-94e2e23fcfd0/volumes"
Oct 31 13:44:02.770282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.770255    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"]
Oct 31 13:44:02.771369 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.771346    4428 cpu_manager.go:414] "containerMap: removing container" podUID="50a90471-68bd-4059-ba8b-8719805293e8" containerName="istio-proxy"
Oct 31 13:44:02.771482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.771372    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a90471-68bd-4059-ba8b-8719805293e8" containerName="istio-proxy"
Oct 31 13:44:02.771994 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.771964    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a90471-68bd-4059-ba8b-8719805293e8" containerName="istio-proxy"
Oct 31 13:44:02.772130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.772004    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a90471-68bd-4059-ba8b-8719805293e8" containerName="istio-proxy"
Oct 31 13:44:02.778745 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.778712    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.817354 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.817317    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q" event={"ID":"db1e5779-e42f-48ed-9804-31af5494ce8a","Type":"ContainerStarted","Data":"a692a5dc099d4126d0fbe8387c8c3eaf6e865c1270522cb0b5fbf2b25256f7c5"}
Oct 31 13:44:02.822444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.822412    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"]
Oct 31 13:44:02.824283 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.824256    4428 generic.go:356] "Generic (PLEG): container finished" podID="50a90471-68bd-4059-ba8b-8719805293e8" containerID="5aa2ef4d9cfa75cdacb5ebe5f9137144e0c359afdf8a1073e0982001b94f07b5" exitCode=0
Oct 31 13:44:02.824430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.824331    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt" event={"ID":"50a90471-68bd-4059-ba8b-8719805293e8","Type":"ContainerDied","Data":"5aa2ef4d9cfa75cdacb5ebe5f9137144e0c359afdf8a1073e0982001b94f07b5"}
Oct 31 13:44:02.824430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.824372    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt" event={"ID":"50a90471-68bd-4059-ba8b-8719805293e8","Type":"ContainerDied","Data":"833ef678252324648d4ef3e7dacff7c34ec78b42a4b4e212a547beb83debcb93"}
Oct 31 13:44:02.824430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.824400    4428 scope.go:122] "RemoveContainer" containerID="5aa2ef4d9cfa75cdacb5ebe5f9137144e0c359afdf8a1073e0982001b94f07b5"
Oct 31 13:44:02.824600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.824538    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"
Oct 31 13:44:02.851330 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.851296    4428 scope.go:122] "RemoveContainer" containerID="ebaf9521a92268d7cd75b8bbaaf944dd891d705d2373851381286fec4dcb6e5d"
Oct 31 13:44:02.855411 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.855318    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.855648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.855476    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.855759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.855675    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93007950-2d7f-495f-a37d-c8729493b79d-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.855759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.855708    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq65m\" (UniqueName: \"kubernetes.io/projected/93007950-2d7f-495f-a37d-c8729493b79d-kube-api-access-bq65m\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.856053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.855824    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.856053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.855964    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.856053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.856007    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.856341 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.856210    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93007950-2d7f-495f-a37d-c8729493b79d-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.856435 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.856378    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93007950-2d7f-495f-a37d-c8729493b79d-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.856619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.856601    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:02.856724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.856624    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:02.856724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.856638    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:02.856724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.856665    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a90471-68bd-4059-ba8b-8719805293e8-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:02.856724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.856681    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a90471-68bd-4059-ba8b-8719805293e8-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:02.856724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.856697    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a90471-68bd-4059-ba8b-8719805293e8-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:02.857100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.856712    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9czv\" (UniqueName: \"kubernetes.io/projected/50a90471-68bd-4059-ba8b-8719805293e8-kube-api-access-f9czv\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:02.857100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.856740    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a90471-68bd-4059-ba8b-8719805293e8-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:02.891177 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.891136    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"]
Oct 31 13:44:02.893910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.893850    4428 scope.go:122] "RemoveContainer" containerID="5aa2ef4d9cfa75cdacb5ebe5f9137144e0c359afdf8a1073e0982001b94f07b5"
Oct 31 13:44:02.895608 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:44:02.895557    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aa2ef4d9cfa75cdacb5ebe5f9137144e0c359afdf8a1073e0982001b94f07b5\": container with ID starting with 5aa2ef4d9cfa75cdacb5ebe5f9137144e0c359afdf8a1073e0982001b94f07b5 not found: ID does not exist" containerID="5aa2ef4d9cfa75cdacb5ebe5f9137144e0c359afdf8a1073e0982001b94f07b5"
Oct 31 13:44:02.895819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.895791    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa2ef4d9cfa75cdacb5ebe5f9137144e0c359afdf8a1073e0982001b94f07b5"} err="failed to get container status \"5aa2ef4d9cfa75cdacb5ebe5f9137144e0c359afdf8a1073e0982001b94f07b5\": rpc error: code = NotFound desc = could not find container \"5aa2ef4d9cfa75cdacb5ebe5f9137144e0c359afdf8a1073e0982001b94f07b5\": container with ID starting with 5aa2ef4d9cfa75cdacb5ebe5f9137144e0c359afdf8a1073e0982001b94f07b5 not found: ID does not exist"
Oct 31 13:44:02.896272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.896232    4428 scope.go:122] "RemoveContainer" containerID="ebaf9521a92268d7cd75b8bbaaf944dd891d705d2373851381286fec4dcb6e5d"
Oct 31 13:44:02.896852 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:44:02.896815    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebaf9521a92268d7cd75b8bbaaf944dd891d705d2373851381286fec4dcb6e5d\": container with ID starting with ebaf9521a92268d7cd75b8bbaaf944dd891d705d2373851381286fec4dcb6e5d not found: ID does not exist" containerID="ebaf9521a92268d7cd75b8bbaaf944dd891d705d2373851381286fec4dcb6e5d"
Oct 31 13:44:02.897128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.897072    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebaf9521a92268d7cd75b8bbaaf944dd891d705d2373851381286fec4dcb6e5d"} err="failed to get container status \"ebaf9521a92268d7cd75b8bbaaf944dd891d705d2373851381286fec4dcb6e5d\": rpc error: code = NotFound desc = could not find container \"ebaf9521a92268d7cd75b8bbaaf944dd891d705d2373851381286fec4dcb6e5d\": container with ID starting with ebaf9521a92268d7cd75b8bbaaf944dd891d705d2373851381286fec4dcb6e5d not found: ID does not exist"
Oct 31 13:44:02.903249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.903225    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-g9lwt"]
Oct 31 13:44:02.976906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.969553    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.976906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.969026    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.976906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.973009    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.976906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.973062    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.976906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.973198    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93007950-2d7f-495f-a37d-c8729493b79d-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.976906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.973288    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93007950-2d7f-495f-a37d-c8729493b79d-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.976906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.973357    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.976906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.973451    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.976906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.973602    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93007950-2d7f-495f-a37d-c8729493b79d-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:02.976906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.973627    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq65m\" (UniqueName: \"kubernetes.io/projected/93007950-2d7f-495f-a37d-c8729493b79d-kube-api-access-bq65m\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:03.023599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.977657    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93007950-2d7f-495f-a37d-c8729493b79d-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:03.023599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.978015    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:03.023599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.978686    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:03.023599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.979019    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:03.023599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:02.995840    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93007950-2d7f-495f-a37d-c8729493b79d-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:03.023599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:03.012033    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93007950-2d7f-495f-a37d-c8729493b79d-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:03.023599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:03.013203    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq65m\" (UniqueName: \"kubernetes.io/projected/93007950-2d7f-495f-a37d-c8729493b79d-kube-api-access-bq65m\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:03.023599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:03.023510    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:03.036367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:03.036297    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:03.114289 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:03.114190    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:03.439801 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:44:03.439764    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93007950_2d7f_495f_a37d_c8729493b79d.slice/crio-bcd7791a5aed6ab1b122797c6ae22acf7119344bd57cd0d8fa78eedc77be77ff WatchSource:0}: Error finding container bcd7791a5aed6ab1b122797c6ae22acf7119344bd57cd0d8fa78eedc77be77ff: Status 404 returned error can't find the container with id bcd7791a5aed6ab1b122797c6ae22acf7119344bd57cd0d8fa78eedc77be77ff
Oct 31 13:44:03.440000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:03.439813    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"]
Oct 31 13:44:03.443037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:03.443003    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:44:03.443145 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:03.443067    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:44:03.443145 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:03.443101    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:44:03.835744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:03.835700    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q" event={"ID":"db1e5779-e42f-48ed-9804-31af5494ce8a","Type":"ContainerStarted","Data":"5c8846842f03cd5ac360c832a2cd850d6339161b69503afb5f5fef2850a22242"}
Oct 31 13:44:03.838105 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:03.838075    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6" event={"ID":"93007950-2d7f-495f-a37d-c8729493b79d","Type":"ContainerStarted","Data":"1d0fc13c282706c183153e71f2b0ac0c3e7c9183b0455f5df2f40aca13d6f89d"}
Oct 31 13:44:03.838273 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:03.838131    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6" event={"ID":"93007950-2d7f-495f-a37d-c8729493b79d","Type":"ContainerStarted","Data":"bcd7791a5aed6ab1b122797c6ae22acf7119344bd57cd0d8fa78eedc77be77ff"}
Oct 31 13:44:03.924907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:03.922389    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:03.942698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:03.942541    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6" podStartSLOduration=262.942518425 podStartE2EDuration="4m22.942518425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:44:03.929996251 +0000 UTC m=+2441.328344877" watchObservedRunningTime="2026-10-31 13:44:03.942518425 +0000 UTC m=+2441.340867049"
Oct 31 13:44:03.952908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:03.946976    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q" podStartSLOduration=41.946956056 podStartE2EDuration="41.946956056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:44:03.885250705 +0000 UTC m=+2441.283599374" watchObservedRunningTime="2026-10-31 13:44:03.946956056 +0000 UTC m=+2441.345304680"
Oct 31 13:44:04.115214 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:04.115128    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:04.117157 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:04.117126    4428 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 13:44:04.117319 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:04.117182    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6" podUID="93007950-2d7f-495f-a37d-c8729493b79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 13:44:04.375957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:04.365528    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:04.375957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:04.372693    4428 patch_prober.go:29] interesting pod/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 13:44:04.375957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:04.372774    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q" podUID="db1e5779-e42f-48ed-9804-31af5494ce8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 13:44:04.375957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:04.374662    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d" podUID="b83d1055-13eb-489e-9bc2-17780894bcfb" containerName="payload-pre-processing" containerID="cri-o://516b4378255b44897cd9ea1f5ee64be6705eb738e009eb0fc2f232281ff4ad41" gracePeriod=25
Oct 31 13:44:04.384197 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:04.384151    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s" podUID="371c7f96-669c-455b-a063-29303ee30cb2" containerName="payload-processing" containerID="cri-o://9999772c1ecb24b4f956e4cb881e244522290f2edc77289f0f17888bd6868b84" gracePeriod=25
Oct 31 13:44:04.803489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:04.803452    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a90471-68bd-4059-ba8b-8719805293e8" path="/var/lib/kubelet/pods/50a90471-68bd-4059-ba8b-8719805293e8/volumes"
Oct 31 13:44:05.062113 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:05.062016    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d" podUID="b83d1055-13eb-489e-9bc2-17780894bcfb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:44:05.067980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:05.067913    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:05.115888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:05.115827    4428 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 13:44:05.116215 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:05.116169    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6" podUID="93007950-2d7f-495f-a37d-c8729493b79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 13:44:05.380537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:05.364932    4428 patch_prober.go:29] interesting pod/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 13:44:05.380537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:05.365029    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q" podUID="db1e5779-e42f-48ed-9804-31af5494ce8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 13:44:06.047900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:06.034172    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:06.118002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:06.117962    4428 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 13:44:06.118625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:06.118589    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6" podUID="93007950-2d7f-495f-a37d-c8729493b79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 13:44:06.369893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:06.349311    4428 patch_prober.go:29] interesting pod/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 13:44:06.369893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:06.349377    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q" podUID="db1e5779-e42f-48ed-9804-31af5494ce8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 13:44:06.941891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:06.940490    4428 generic.go:356] "Generic (PLEG): container finished" podID="37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" containerID="00f748be89a81f9ff06efc0c3f850515ca8fe06d72b33d04e931819ae67f8429" exitCode=0
Oct 31 13:44:06.941891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:06.940709    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6" event={"ID":"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4","Type":"ContainerDied","Data":"00f748be89a81f9ff06efc0c3f850515ca8fe06d72b33d04e931819ae67f8429"}
Oct 31 13:44:07.112933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.111637    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:07.123907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.120428    4428 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 13:44:07.123907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.120504    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6" podUID="93007950-2d7f-495f-a37d-c8729493b79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 13:44:07.131907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.129184    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:44:07.283191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.277920    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-workload-socket\") pod \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") "
Oct 31 13:44:07.283191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.277971    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-credential-socket\") pod \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") "
Oct 31 13:44:07.283191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.278042    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-data\") pod \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") "
Oct 31 13:44:07.283191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.278081    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-podinfo\") pod \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") "
Oct 31 13:44:07.283191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.278274    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-envoy\") pod \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") "
Oct 31 13:44:07.283191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.278463    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-workload-certs\") pod \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") "
Oct 31 13:44:07.283191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.278535    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istiod-ca-cert\") pod \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") "
Oct 31 13:44:07.283191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.278602    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-token\" (UniqueName: \"kubernetes.io/projected/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-token\") pod \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") "
Oct 31 13:44:07.283191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.278659    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-kube-api-access-84bf9\" (UniqueName: \"kubernetes.io/projected/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-kube-api-access-84bf9\") pod \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\" (UID: \"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4\") "
Oct 31 13:44:07.309118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.292686    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istiod-ca-cert" pod "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" (UID: "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:07.309118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.301188    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-credential-socket" pod "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" (UID: "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:07.309118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.301322    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-workload-socket" pod "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" (UID: "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:07.309118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.301500    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-data" pod "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" (UID: "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:07.309118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.301786    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-workload-certs" pod "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" (UID: "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:07.348922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.348428    4428 patch_prober.go:29] interesting pod/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 13:44:07.348922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.348496    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q" podUID="db1e5779-e42f-48ed-9804-31af5494ce8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 13:44:07.393919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.379409    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-token" pod "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" (UID: "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:07.393919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.379541    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-envoy" pod "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" (UID: "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:07.393919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.383205    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:07.393919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.383238    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:07.393919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.383253    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:07.393919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.383267    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:07.393919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.383280    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:07.393919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.383343    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:07.393919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.383360    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:07.407913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.407783    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-kube-api-access-84bf9" pod "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" (UID: "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4"). InnerVolumeSpecName "kube-api-access-84bf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:07.425809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.424373    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-podinfo" pod "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" (UID: "37cf573f-d0ac-4fa3-bec6-24715bfdb9b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:44:07.500893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.494699    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84bf9\" (UniqueName: \"kubernetes.io/projected/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-kube-api-access-84bf9\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:07.500893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.494738    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:07.978028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.977987    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6" event={"ID":"37cf573f-d0ac-4fa3-bec6-24715bfdb9b4","Type":"ContainerDied","Data":"edd2a2313808476012738af4f6c2870681528d50611f75b45552f1c951012dda"}
Oct 31 13:44:07.978316 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.978296    4428 scope.go:122] "RemoveContainer" containerID="00f748be89a81f9ff06efc0c3f850515ca8fe06d72b33d04e931819ae67f8429"
Oct 31 13:44:07.978643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:07.978627    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"
Oct 31 13:44:08.122344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:08.122299    4428 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 13:44:08.122639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:08.122604    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6" podUID="93007950-2d7f-495f-a37d-c8729493b79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 13:44:08.128809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:08.128771    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"]
Oct 31 13:44:08.154467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:08.154410    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:08.249961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:08.244889    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-dkvh6"]
Oct 31 13:44:08.364894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:08.349272    4428 patch_prober.go:29] interesting pod/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 13:44:08.364894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:08.349337    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q" podUID="db1e5779-e42f-48ed-9804-31af5494ce8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 13:44:08.808692 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:08.808659    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" path="/var/lib/kubelet/pods/37cf573f-d0ac-4fa3-bec6-24715bfdb9b4/volumes"
Oct 31 13:44:09.115622 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:09.115540    4428 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 13:44:09.115622 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:09.115589    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6" podUID="93007950-2d7f-495f-a37d-c8729493b79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 13:44:09.288734 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:09.288665    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:09.355597 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:09.355562    4428 patch_prober.go:29] interesting pod/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 13:44:09.355903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:09.355856    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q" podUID="db1e5779-e42f-48ed-9804-31af5494ce8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 13:44:09.996242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:09.996124    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:10.115317 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:10.115269    4428 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 13:44:10.115512 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:10.115356    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6" podUID="93007950-2d7f-495f-a37d-c8729493b79d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 13:44:10.258111 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:10.257808    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:10.352907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:10.351320    4428 patch_prober.go:29] interesting pod/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 13:44:10.352907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:10.351402    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q" podUID="db1e5779-e42f-48ed-9804-31af5494ce8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 13:44:10.518607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:10.518503    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:11.122608 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.122571    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:11.123147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.123124    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:11.123930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.123892    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:44:11.252756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.252715    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:11.300074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.300041    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"]
Oct 31 13:44:11.301224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.301198    4428 cpu_manager.go:414] "containerMap: removing container" podUID="50a90471-68bd-4059-ba8b-8719805293e8" containerName="istio-proxy"
Oct 31 13:44:11.301224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.301220    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a90471-68bd-4059-ba8b-8719805293e8" containerName="istio-proxy"
Oct 31 13:44:11.301422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.301241    4428 cpu_manager.go:414] "containerMap: removing container" podUID="37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" containerName="istio-proxy"
Oct 31 13:44:11.301422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.301252    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" containerName="istio-proxy"
Oct 31 13:44:11.301539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.301521    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="37cf573f-d0ac-4fa3-bec6-24715bfdb9b4" containerName="istio-proxy"
Oct 31 13:44:11.307255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.307224    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"
Oct 31 13:44:11.316675 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.316646    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-72c210\""
Oct 31 13:44:11.320011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.319985    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"]
Oct 31 13:44:11.326432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.326409    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"]
Oct 31 13:44:11.326559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.326521    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"
Oct 31 13:44:11.345316 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.345290    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"]
Oct 31 13:44:11.351924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.351898    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-72c210-dockercfg-ttmls\""
Oct 31 13:44:11.357093 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.357058    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:11.357539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.357515    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:11.358684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.358660    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:44:11.473918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.472223    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803c3db4-5311-4a96-8474-64304a5995ba-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs\" (UID: \"803c3db4-5311-4a96-8474-64304a5995ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"
Oct 31 13:44:11.473918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.472337    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kjsk\" (UniqueName: \"kubernetes.io/projected/1821cf0c-d453-4c5c-83bd-7c3e22666953-kube-api-access-5kjsk\") pod \"payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2\" (UID: \"1821cf0c-d453-4c5c-83bd-7c3e22666953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"
Oct 31 13:44:11.473918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.472449    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj9xr\" (UniqueName: \"kubernetes.io/projected/803c3db4-5311-4a96-8474-64304a5995ba-kube-api-access-wj9xr\") pod \"payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs\" (UID: \"803c3db4-5311-4a96-8474-64304a5995ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"
Oct 31 13:44:11.473918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.472558    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1821cf0c-d453-4c5c-83bd-7c3e22666953-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2\" (UID: \"1821cf0c-d453-4c5c-83bd-7c3e22666953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"
Oct 31 13:44:11.777954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.771432    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803c3db4-5311-4a96-8474-64304a5995ba-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs\" (UID: \"803c3db4-5311-4a96-8474-64304a5995ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"
Oct 31 13:44:11.777954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.771539    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kjsk\" (UniqueName: \"kubernetes.io/projected/1821cf0c-d453-4c5c-83bd-7c3e22666953-kube-api-access-5kjsk\") pod \"payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2\" (UID: \"1821cf0c-d453-4c5c-83bd-7c3e22666953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"
Oct 31 13:44:11.777954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.771671    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj9xr\" (UniqueName: \"kubernetes.io/projected/803c3db4-5311-4a96-8474-64304a5995ba-kube-api-access-wj9xr\") pod \"payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs\" (UID: \"803c3db4-5311-4a96-8474-64304a5995ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"
Oct 31 13:44:11.777954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.771781    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1821cf0c-d453-4c5c-83bd-7c3e22666953-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2\" (UID: \"1821cf0c-d453-4c5c-83bd-7c3e22666953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"
Oct 31 13:44:11.849898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.841905    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803c3db4-5311-4a96-8474-64304a5995ba-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs\" (UID: \"803c3db4-5311-4a96-8474-64304a5995ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"
Oct 31 13:44:11.849898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.845852    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:11.857010 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.850550    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1821cf0c-d453-4c5c-83bd-7c3e22666953-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2\" (UID: \"1821cf0c-d453-4c5c-83bd-7c3e22666953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"
Oct 31 13:44:11.897900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.897551    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kjsk\" (UniqueName: \"kubernetes.io/projected/1821cf0c-d453-4c5c-83bd-7c3e22666953-kube-api-access-5kjsk\") pod \"payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2\" (UID: \"1821cf0c-d453-4c5c-83bd-7c3e22666953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"
Oct 31 13:44:11.926937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.926833    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"
Oct 31 13:44:11.954951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:11.954832    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj9xr\" (UniqueName: \"kubernetes.io/projected/803c3db4-5311-4a96-8474-64304a5995ba-kube-api-access-wj9xr\") pod \"payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs\" (UID: \"803c3db4-5311-4a96-8474-64304a5995ba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"
Oct 31 13:44:12.092393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:12.092358    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:12.247780 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:12.247747    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"
Oct 31 13:44:12.293179 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:44:12.293137    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1821cf0c_d453_4c5c_83bd_7c3e22666953.slice/crio-a775b1a12b81d7be3ee631c56df79732feabbad02eed7f23fdcc4a76d23e83ac WatchSource:0}: Error finding container a775b1a12b81d7be3ee631c56df79732feabbad02eed7f23fdcc4a76d23e83ac: Status 404 returned error can't find the container with id a775b1a12b81d7be3ee631c56df79732feabbad02eed7f23fdcc4a76d23e83ac
Oct 31 13:44:12.294835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:12.294808    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"]
Oct 31 13:44:12.584957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:12.579350    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:12.613895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:12.604292    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"]
Oct 31 13:44:13.057060 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:13.057022    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs" event={"ID":"803c3db4-5311-4a96-8474-64304a5995ba","Type":"ContainerStarted","Data":"08b0b8a391e9cace2c0e343492a58fa67eadacbcda35099398054aa402e1ec05"}
Oct 31 13:44:13.057252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:13.057094    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs" event={"ID":"803c3db4-5311-4a96-8474-64304a5995ba","Type":"ContainerStarted","Data":"90934be146edb9eb4cd191c4f0bf07d7354adc146e1affebc4669ce507624d09"}
Oct 31 13:44:13.057252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:13.057121    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"
Oct 31 13:44:13.058962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:13.058933    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2" event={"ID":"1821cf0c-d453-4c5c-83bd-7c3e22666953","Type":"ContainerStarted","Data":"f27596d445e50248b653b6f314d8a86ceea2c0c4896d14253ac5dc4dfca5f2fa"}
Oct 31 13:44:13.059099 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:13.058985    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2" event={"ID":"1821cf0c-d453-4c5c-83bd-7c3e22666953","Type":"ContainerStarted","Data":"a775b1a12b81d7be3ee631c56df79732feabbad02eed7f23fdcc4a76d23e83ac"}
Oct 31 13:44:13.084055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:13.083997    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs" podStartSLOduration=2.083981763 podStartE2EDuration="2.083981763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:44:13.078142902 +0000 UTC m=+2450.476491528" watchObservedRunningTime="2026-10-31 13:44:13.083981763 +0000 UTC m=+2450.482330387"
Oct 31 13:44:13.100567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:13.100501    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2" podStartSLOduration=2.100482884 podStartE2EDuration="2.100482884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:44:13.09875472 +0000 UTC m=+2450.497103345" watchObservedRunningTime="2026-10-31 13:44:13.100482884 +0000 UTC m=+2450.498831509"
Oct 31 13:44:13.633904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:13.633345    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:14.029848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:14.029806    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s" podUID="371c7f96-669c-455b-a063-29303ee30cb2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:44:14.067399 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:14.067369    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"
Oct 31 13:44:14.182196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:14.182165    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:14.543692 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:14.543275    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:14.939293 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:14.939206    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:15.040146 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:15.040107    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d" podUID="b83d1055-13eb-489e-9bc2-17780894bcfb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:44:15.683717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:15.683682    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:16.007690 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:16.007628    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"]
Oct 31 13:44:16.008460 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:16.008263    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl" podUID="7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" containerName="istio-proxy" containerID="cri-o://48e627edfa7d2dfb092e316b6e688a7f7f5b3bd135cdde5ee01a6d34334adb83" gracePeriod=30
Oct 31 13:44:16.752891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:16.747762    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:17.733251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:17.733218    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:18.799335 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:44:18.799296    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:44:18.801494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:18.801461    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:19.779849 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:19.779805    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:20.914984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:20.914941    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:21.163539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:21.163497    4428 generic.go:356] "Generic (PLEG): container finished" podID="7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" containerID="48e627edfa7d2dfb092e316b6e688a7f7f5b3bd135cdde5ee01a6d34334adb83" exitCode=0
Oct 31 13:44:21.164031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:21.163582    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl" event={"ID":"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6","Type":"ContainerDied","Data":"48e627edfa7d2dfb092e316b6e688a7f7f5b3bd135cdde5ee01a6d34334adb83"}
Oct 31 13:44:21.829045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:21.828966    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:44:21.842566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:21.842521    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:21.987895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:21.986545    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-workload-certs\") pod \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") "
Oct 31 13:44:21.987895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:21.986663    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-envoy\") pod \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") "
Oct 31 13:44:21.987895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:21.986710    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-data\") pod \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") "
Oct 31 13:44:21.987895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:21.986771    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-workload-socket\") pod \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") "
Oct 31 13:44:21.987895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:21.986855    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-podinfo\") pod \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") "
Oct 31 13:44:21.987895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:21.986922    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-kube-api-access-q8ggx\" (UniqueName: \"kubernetes.io/projected/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-kube-api-access-q8ggx\") pod \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") "
Oct 31 13:44:21.987895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:21.987155    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-credential-socket\") pod \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") "
Oct 31 13:44:21.987895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:21.987207    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-token\" (UniqueName: \"kubernetes.io/projected/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-token\") pod \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") "
Oct 31 13:44:21.987895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:21.987307    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istiod-ca-cert\") pod \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\" (UID: \"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6\") "
Oct 31 13:44:22.017946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.002624    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istiod-ca-cert" pod "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" (UID: "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:22.017946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.002966    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-workload-certs" pod "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" (UID: "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:22.017946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.013024    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-credential-socket" pod "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" (UID: "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:22.017946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.014405    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-podinfo" pod "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" (UID: "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:44:22.036892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.018338    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-data" pod "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" (UID: "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:22.036892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.018612    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-workload-socket" pod "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" (UID: "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:22.036892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.019536    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-envoy" pod "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" (UID: "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:22.036892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.023037    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-kube-api-access-q8ggx" pod "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" (UID: "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6"). InnerVolumeSpecName "kube-api-access-q8ggx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:22.075891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.075021    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-token" pod "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" (UID: "7bdaf4ff-dca2-4955-bdd7-d3309ef783c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:22.104412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.103900    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:22.104412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.103937    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:22.104412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.103955    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:22.104412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.103971    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8ggx\" (UniqueName: \"kubernetes.io/projected/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-kube-api-access-q8ggx\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:22.104412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.103988    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:22.104412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.104004    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:22.104412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.104029    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:22.104412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.104043    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:22.104412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.104057    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:22.174369 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.174331    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl" event={"ID":"7bdaf4ff-dca2-4955-bdd7-d3309ef783c6","Type":"ContainerDied","Data":"9d610183403a6f7e700fb8759af11c6306f8bf8d2e7015e3b407ae263954b6b4"}
Oct 31 13:44:22.174573 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.174402    4428 scope.go:122] "RemoveContainer" containerID="48e627edfa7d2dfb092e316b6e688a7f7f5b3bd135cdde5ee01a6d34334adb83"
Oct 31 13:44:22.174573 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.174402    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"
Oct 31 13:44:22.206637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.206609    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"]
Oct 31 13:44:22.214884 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.214843    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-sbnsl"]
Oct 31 13:44:22.406104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.406009    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:22.515853 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.515815    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"]
Oct 31 13:44:22.516147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.516119    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd" podUID="c6533b62-cafe-4af5-a32d-2bb882ebe41b" containerName="maas-api" containerID="cri-o://600f760be36d4e5925fa614d63f3a39abf47196fd01ba82f8787d2c2f21ea4f0" gracePeriod=30
Oct 31 13:44:22.582855 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.582819    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:22.761769 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.761734    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" path="/var/lib/kubelet/pods/7bdaf4ff-dca2-4955-bdd7-d3309ef783c6/volumes"
Oct 31 13:44:22.792521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.792487    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:22.839014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.837711    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:44:22.938892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.934999    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6533b62-cafe-4af5-a32d-2bb882ebe41b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6533b62-cafe-4af5-a32d-2bb882ebe41b-maas-api-tls\") pod \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\" (UID: \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\") "
Oct 31 13:44:22.938892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.935123    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6533b62-cafe-4af5-a32d-2bb882ebe41b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6533b62-cafe-4af5-a32d-2bb882ebe41b-metrics-certs\") pod \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\" (UID: \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\") "
Oct 31 13:44:22.938892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.935971    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6533b62-cafe-4af5-a32d-2bb882ebe41b-kube-api-access-z5hzf\" (UniqueName: \"kubernetes.io/projected/c6533b62-cafe-4af5-a32d-2bb882ebe41b-kube-api-access-z5hzf\") pod \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\" (UID: \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\") "
Oct 31 13:44:22.938892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.936133    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6533b62-cafe-4af5-a32d-2bb882ebe41b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6533b62-cafe-4af5-a32d-2bb882ebe41b-openshift-service-ca\") pod \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\" (UID: \"c6533b62-cafe-4af5-a32d-2bb882ebe41b\") "
Oct 31 13:44:22.987189 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.960109    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6533b62-cafe-4af5-a32d-2bb882ebe41b-openshift-service-ca" pod "c6533b62-cafe-4af5-a32d-2bb882ebe41b" (UID: "c6533b62-cafe-4af5-a32d-2bb882ebe41b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:22.987189 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.983021    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6533b62-cafe-4af5-a32d-2bb882ebe41b-metrics-certs" pod "c6533b62-cafe-4af5-a32d-2bb882ebe41b" (UID: "c6533b62-cafe-4af5-a32d-2bb882ebe41b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:44:22.996894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.992340    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6533b62-cafe-4af5-a32d-2bb882ebe41b-maas-api-tls" pod "c6533b62-cafe-4af5-a32d-2bb882ebe41b" (UID: "c6533b62-cafe-4af5-a32d-2bb882ebe41b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:44:22.998084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:22.998020    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6533b62-cafe-4af5-a32d-2bb882ebe41b-kube-api-access-z5hzf" pod "c6533b62-cafe-4af5-a32d-2bb882ebe41b" (UID: "c6533b62-cafe-4af5-a32d-2bb882ebe41b"). InnerVolumeSpecName "kube-api-access-z5hzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:23.012064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:23.005365    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:23.145734 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:23.095509    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6533b62-cafe-4af5-a32d-2bb882ebe41b-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:23.243342 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:23.243223    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6533b62-cafe-4af5-a32d-2bb882ebe41b-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:23.243582 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:23.243565    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5hzf\" (UniqueName: \"kubernetes.io/projected/c6533b62-cafe-4af5-a32d-2bb882ebe41b-kube-api-access-z5hzf\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:23.243719 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:23.243706    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6533b62-cafe-4af5-a32d-2bb882ebe41b-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:23.243837 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:23.242694    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"
Oct 31 13:44:23.244120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:23.227910    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:23.244445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:23.242594    4428 generic.go:356] "Generic (PLEG): container finished" podID="c6533b62-cafe-4af5-a32d-2bb882ebe41b" containerID="600f760be36d4e5925fa614d63f3a39abf47196fd01ba82f8787d2c2f21ea4f0" exitCode=0
Oct 31 13:44:23.244732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:23.242628    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd" event={"ID":"c6533b62-cafe-4af5-a32d-2bb882ebe41b","Type":"ContainerDied","Data":"600f760be36d4e5925fa614d63f3a39abf47196fd01ba82f8787d2c2f21ea4f0"}
Oct 31 13:44:23.244963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:23.244850    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd" event={"ID":"c6533b62-cafe-4af5-a32d-2bb882ebe41b","Type":"ContainerDied","Data":"571af416397612c1176e48039f027ee606d6372ae54755cd7948fd3516f315d4"}
Oct 31 13:44:23.244963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:23.244934    4428 scope.go:122] "RemoveContainer" containerID="600f760be36d4e5925fa614d63f3a39abf47196fd01ba82f8787d2c2f21ea4f0"
Oct 31 13:44:23.334857 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:23.334827    4428 scope.go:122] "RemoveContainer" containerID="600f760be36d4e5925fa614d63f3a39abf47196fd01ba82f8787d2c2f21ea4f0"
Oct 31 13:44:23.335337 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:44:23.335286    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600f760be36d4e5925fa614d63f3a39abf47196fd01ba82f8787d2c2f21ea4f0\": container with ID starting with 600f760be36d4e5925fa614d63f3a39abf47196fd01ba82f8787d2c2f21ea4f0 not found: ID does not exist" containerID="600f760be36d4e5925fa614d63f3a39abf47196fd01ba82f8787d2c2f21ea4f0"
Oct 31 13:44:23.335478 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:23.335351    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600f760be36d4e5925fa614d63f3a39abf47196fd01ba82f8787d2c2f21ea4f0"} err="failed to get container status \"600f760be36d4e5925fa614d63f3a39abf47196fd01ba82f8787d2c2f21ea4f0\": rpc error: code = NotFound desc = could not find container \"600f760be36d4e5925fa614d63f3a39abf47196fd01ba82f8787d2c2f21ea4f0\": container with ID starting with 600f760be36d4e5925fa614d63f3a39abf47196fd01ba82f8787d2c2f21ea4f0 not found: ID does not exist"
Oct 31 13:44:23.422287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:23.403836    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"]
Oct 31 13:44:23.422287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:23.411317    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69bdf5-844599fd8b-z9rbd"]
Oct 31 13:44:23.460537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:23.459456    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:23.822295 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:23.822258    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:24.028524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:24.028488    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s" podUID="371c7f96-669c-455b-a063-29303ee30cb2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:44:24.070662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:24.070635    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"
Oct 31 13:44:24.386045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:24.386006    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:24.739500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:24.739474    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"
Oct 31 13:44:24.748116 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:24.748087    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"
Oct 31 13:44:24.778125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:24.778093    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6533b62-cafe-4af5-a32d-2bb882ebe41b" path="/var/lib/kubelet/pods/c6533b62-cafe-4af5-a32d-2bb882ebe41b/volumes"
Oct 31 13:44:24.802134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:24.802096    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371c7f96-669c-455b-a063-29303ee30cb2-kube-api-access-pcrr4\" (UniqueName: \"kubernetes.io/projected/371c7f96-669c-455b-a063-29303ee30cb2-kube-api-access-pcrr4\") pod \"371c7f96-669c-455b-a063-29303ee30cb2\" (UID: \"371c7f96-669c-455b-a063-29303ee30cb2\") "
Oct 31 13:44:24.802846 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:24.802820    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371c7f96-669c-455b-a063-29303ee30cb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371c7f96-669c-455b-a063-29303ee30cb2-plugins-config-volume\") pod \"371c7f96-669c-455b-a063-29303ee30cb2\" (UID: \"371c7f96-669c-455b-a063-29303ee30cb2\") "
Oct 31 13:44:24.803770 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:24.803739    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371c7f96-669c-455b-a063-29303ee30cb2-plugins-config-volume" pod "371c7f96-669c-455b-a063-29303ee30cb2" (UID: "371c7f96-669c-455b-a063-29303ee30cb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:24.806099 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:24.806069    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371c7f96-669c-455b-a063-29303ee30cb2-kube-api-access-pcrr4" pod "371c7f96-669c-455b-a063-29303ee30cb2" (UID: "371c7f96-669c-455b-a063-29303ee30cb2"). InnerVolumeSpecName "kube-api-access-pcrr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:24.904610 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:24.904567    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b83d1055-13eb-489e-9bc2-17780894bcfb-kube-api-access-fj7kp\" (UniqueName: \"kubernetes.io/projected/b83d1055-13eb-489e-9bc2-17780894bcfb-kube-api-access-fj7kp\") pod \"b83d1055-13eb-489e-9bc2-17780894bcfb\" (UID: \"b83d1055-13eb-489e-9bc2-17780894bcfb\") "
Oct 31 13:44:24.904831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:24.904797    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b83d1055-13eb-489e-9bc2-17780894bcfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83d1055-13eb-489e-9bc2-17780894bcfb-plugins-config-volume\") pod \"b83d1055-13eb-489e-9bc2-17780894bcfb\" (UID: \"b83d1055-13eb-489e-9bc2-17780894bcfb\") "
Oct 31 13:44:24.905249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:24.905221    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83d1055-13eb-489e-9bc2-17780894bcfb-plugins-config-volume" pod "b83d1055-13eb-489e-9bc2-17780894bcfb" (UID: "b83d1055-13eb-489e-9bc2-17780894bcfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:24.905813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:24.905791    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcrr4\" (UniqueName: \"kubernetes.io/projected/371c7f96-669c-455b-a063-29303ee30cb2-kube-api-access-pcrr4\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:24.905978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:24.905819    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83d1055-13eb-489e-9bc2-17780894bcfb-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:24.905978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:24.905835    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371c7f96-669c-455b-a063-29303ee30cb2-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:24.907084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:24.907064    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83d1055-13eb-489e-9bc2-17780894bcfb-kube-api-access-fj7kp" pod "b83d1055-13eb-489e-9bc2-17780894bcfb" (UID: "b83d1055-13eb-489e-9bc2-17780894bcfb"). InnerVolumeSpecName "kube-api-access-fj7kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:25.007609 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.007561    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj7kp\" (UniqueName: \"kubernetes.io/projected/b83d1055-13eb-489e-9bc2-17780894bcfb-kube-api-access-fj7kp\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:25.080485 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.080454    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"
Oct 31 13:44:25.267315 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.267219    4428 generic.go:356] "Generic (PLEG): container finished" podID="b83d1055-13eb-489e-9bc2-17780894bcfb" containerID="516b4378255b44897cd9ea1f5ee64be6705eb738e009eb0fc2f232281ff4ad41" exitCode=0
Oct 31 13:44:25.267315 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.267289    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"
Oct 31 13:44:25.267581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.267304    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d" event={"ID":"b83d1055-13eb-489e-9bc2-17780894bcfb","Type":"ContainerDied","Data":"516b4378255b44897cd9ea1f5ee64be6705eb738e009eb0fc2f232281ff4ad41"}
Oct 31 13:44:25.267581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.267367    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d" event={"ID":"b83d1055-13eb-489e-9bc2-17780894bcfb","Type":"ContainerDied","Data":"e6bf00259f76b1cf4a2389df3ad19a54c6780059847fc989eea4e8d57ce9502f"}
Oct 31 13:44:25.267581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.267399    4428 scope.go:122] "RemoveContainer" containerID="516b4378255b44897cd9ea1f5ee64be6705eb738e009eb0fc2f232281ff4ad41"
Oct 31 13:44:25.269191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.269167    4428 generic.go:356] "Generic (PLEG): container finished" podID="371c7f96-669c-455b-a063-29303ee30cb2" containerID="9999772c1ecb24b4f956e4cb881e244522290f2edc77289f0f17888bd6868b84" exitCode=0
Oct 31 13:44:25.269321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.269246    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"
Oct 31 13:44:25.269321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.269249    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s" event={"ID":"371c7f96-669c-455b-a063-29303ee30cb2","Type":"ContainerDied","Data":"9999772c1ecb24b4f956e4cb881e244522290f2edc77289f0f17888bd6868b84"}
Oct 31 13:44:25.269449 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.269356    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s" event={"ID":"371c7f96-669c-455b-a063-29303ee30cb2","Type":"ContainerDied","Data":"7b224fca0a6c95881d4c72788ccdcc8f608e6b5de07831d888ee7d0a66d72809"}
Oct 31 13:44:25.281529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.281510    4428 scope.go:122] "RemoveContainer" containerID="516b4378255b44897cd9ea1f5ee64be6705eb738e009eb0fc2f232281ff4ad41"
Oct 31 13:44:25.281785 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:44:25.281761    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516b4378255b44897cd9ea1f5ee64be6705eb738e009eb0fc2f232281ff4ad41\": container with ID starting with 516b4378255b44897cd9ea1f5ee64be6705eb738e009eb0fc2f232281ff4ad41 not found: ID does not exist" containerID="516b4378255b44897cd9ea1f5ee64be6705eb738e009eb0fc2f232281ff4ad41"
Oct 31 13:44:25.281901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.281791    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516b4378255b44897cd9ea1f5ee64be6705eb738e009eb0fc2f232281ff4ad41"} err="failed to get container status \"516b4378255b44897cd9ea1f5ee64be6705eb738e009eb0fc2f232281ff4ad41\": rpc error: code = NotFound desc = could not find container \"516b4378255b44897cd9ea1f5ee64be6705eb738e009eb0fc2f232281ff4ad41\": container with ID starting with 516b4378255b44897cd9ea1f5ee64be6705eb738e009eb0fc2f232281ff4ad41 not found: ID does not exist"
Oct 31 13:44:25.281901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.281809    4428 scope.go:122] "RemoveContainer" containerID="9999772c1ecb24b4f956e4cb881e244522290f2edc77289f0f17888bd6868b84"
Oct 31 13:44:25.293896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.293860    4428 scope.go:122] "RemoveContainer" containerID="9999772c1ecb24b4f956e4cb881e244522290f2edc77289f0f17888bd6868b84"
Oct 31 13:44:25.294207 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:44:25.294184    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9999772c1ecb24b4f956e4cb881e244522290f2edc77289f0f17888bd6868b84\": container with ID starting with 9999772c1ecb24b4f956e4cb881e244522290f2edc77289f0f17888bd6868b84 not found: ID does not exist" containerID="9999772c1ecb24b4f956e4cb881e244522290f2edc77289f0f17888bd6868b84"
Oct 31 13:44:25.294319 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.294212    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9999772c1ecb24b4f956e4cb881e244522290f2edc77289f0f17888bd6868b84"} err="failed to get container status \"9999772c1ecb24b4f956e4cb881e244522290f2edc77289f0f17888bd6868b84\": rpc error: code = NotFound desc = could not find container \"9999772c1ecb24b4f956e4cb881e244522290f2edc77289f0f17888bd6868b84\": container with ID starting with 9999772c1ecb24b4f956e4cb881e244522290f2edc77289f0f17888bd6868b84 not found: ID does not exist"
Oct 31 13:44:25.301502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.301476    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"]
Oct 31 13:44:25.306511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.306484    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69bdf5-849f4d78cf-shr6d"]
Oct 31 13:44:25.327577 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.327549    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"]
Oct 31 13:44:25.332190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.332165    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69bdf5-b5b8877df-knk8s"]
Oct 31 13:44:25.454961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:25.448726    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:26.482368 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:26.481319    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:26.758797 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:26.758710    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371c7f96-669c-455b-a063-29303ee30cb2" path="/var/lib/kubelet/pods/371c7f96-669c-455b-a063-29303ee30cb2/volumes"
Oct 31 13:44:26.759281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:26.759266    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b83d1055-13eb-489e-9bc2-17780894bcfb" path="/var/lib/kubelet/pods/b83d1055-13eb-489e-9bc2-17780894bcfb/volumes"
Oct 31 13:44:28.029795 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:28.029762    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:28.213815 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:28.213749    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:28.963899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:28.963524    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:29.770893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:29.762501    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:30.634895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:30.630912    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:31.520619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:31.520582    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:32.123750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.123707    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:32.440841 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.440749    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:32.507278 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.507238    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"]
Oct 31 13:44:32.507606 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.507576    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x" podUID="c080a436-0943-4c33-938c-35b1ff2c1a11" containerName="istio-proxy" containerID="cri-o://c6699006304b01ee77cdd77437b3100340400b010222ebb90ddbb9343f322c9b" gracePeriod=30
Oct 31 13:44:32.639895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.639427    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:32.988832 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.988787    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"]
Oct 31 13:44:32.990534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.990503    4428 cpu_manager.go:414] "containerMap: removing container" podUID="371c7f96-669c-455b-a063-29303ee30cb2" containerName="payload-processing"
Oct 31 13:44:32.990704 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.990692    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371c7f96-669c-455b-a063-29303ee30cb2" containerName="payload-processing"
Oct 31 13:44:32.990801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.990792    4428 cpu_manager.go:414] "containerMap: removing container" podUID="c6533b62-cafe-4af5-a32d-2bb882ebe41b" containerName="maas-api"
Oct 31 13:44:32.990896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.990884    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6533b62-cafe-4af5-a32d-2bb882ebe41b" containerName="maas-api"
Oct 31 13:44:32.991041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.991029    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b83d1055-13eb-489e-9bc2-17780894bcfb" containerName="payload-pre-processing"
Oct 31 13:44:32.991123 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.991111    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83d1055-13eb-489e-9bc2-17780894bcfb" containerName="payload-pre-processing"
Oct 31 13:44:32.991245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.991234    4428 cpu_manager.go:414] "containerMap: removing container" podUID="7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" containerName="istio-proxy"
Oct 31 13:44:32.991325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.991315    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" containerName="istio-proxy"
Oct 31 13:44:32.991663 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.991648    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bdaf4ff-dca2-4955-bdd7-d3309ef783c6" containerName="istio-proxy"
Oct 31 13:44:32.991779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.991769    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="371c7f96-669c-455b-a063-29303ee30cb2" containerName="payload-processing"
Oct 31 13:44:32.991887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.991857    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6533b62-cafe-4af5-a32d-2bb882ebe41b" containerName="maas-api"
Oct 31 13:44:32.991983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:32.991973    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83d1055-13eb-489e-9bc2-17780894bcfb" containerName="payload-pre-processing"
Oct 31 13:44:33.010118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:33.010086    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"
Oct 31 13:44:33.016077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:33.015993    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"]
Oct 31 13:44:33.028227 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:33.023317    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:44:33.028227 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:33.023403    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:44:33.093662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:33.093618    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdzcm\" (UniqueName: \"kubernetes.io/projected/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-kube-api-access-fdzcm\") pod \"maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv\" (UID: \"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"
Oct 31 13:44:33.093920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:33.093774    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv\" (UID: \"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"
Oct 31 13:44:33.093984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:33.093968    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv\" (UID: \"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"
Oct 31 13:44:33.195450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:33.195406    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdzcm\" (UniqueName: \"kubernetes.io/projected/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-kube-api-access-fdzcm\") pod \"maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv\" (UID: \"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"
Oct 31 13:44:33.195662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:33.195551    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv\" (UID: \"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"
Oct 31 13:44:33.195767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:33.195749    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv\" (UID: \"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"
Oct 31 13:44:33.196267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:33.196245    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv\" (UID: \"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"
Oct 31 13:44:33.198238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:33.198214    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv\" (UID: \"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"
Oct 31 13:44:33.209318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:33.209293    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdzcm\" (UniqueName: \"kubernetes.io/projected/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-kube-api-access-fdzcm\") pod \"maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv\" (UID: \"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"
Oct 31 13:44:33.351268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:33.351186    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"
Oct 31 13:44:33.528102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:33.527933    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:33.692498 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:33.692469    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"]
Oct 31 13:44:33.692836 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:44:33.692526    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f7fe62_925f_48d0_a15e_ed6c4eb56f2b.slice/crio-9f503edd1f9beb5f08c517d606546b42c2bce7bc148d0443b605ef2ec734abbb WatchSource:0}: Error finding container 9f503edd1f9beb5f08c517d606546b42c2bce7bc148d0443b605ef2ec734abbb: Status 404 returned error can't find the container with id 9f503edd1f9beb5f08c517d606546b42c2bce7bc148d0443b605ef2ec734abbb
Oct 31 13:44:33.773582 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:33.773551    4428 scope.go:122] "RemoveContainer" containerID="ec8d1e4ced2a227d8923e0abd4823f7c9bf59714a79d3fc849e884670c75edea"
Oct 31 13:44:34.368008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:34.367913    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv" event={"ID":"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b","Type":"ContainerStarted","Data":"bf04cb843d2f5a6404f855500c246184f6fe4a4d1248ad41576714064e188ba6"}
Oct 31 13:44:34.368008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:34.367984    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv" event={"ID":"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b","Type":"ContainerStarted","Data":"9f503edd1f9beb5f08c517d606546b42c2bce7bc148d0443b605ef2ec734abbb"}
Oct 31 13:44:34.393415 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:34.393358    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv" podStartSLOduration=2.393341534 podStartE2EDuration="2.393341534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:44:34.389123053 +0000 UTC m=+2471.787471678" watchObservedRunningTime="2026-10-31 13:44:34.393341534 +0000 UTC m=+2471.791690158"
Oct 31 13:44:34.539680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:34.539641    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:35.277502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.277464    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"]
Oct 31 13:44:35.283932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.283855    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:44:35.293184 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.293150    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-72c210-serving-cert\""
Oct 31 13:44:35.313556 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.313521    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"]
Oct 31 13:44:35.357057 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.357019    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1a979ed-3892-4eb5-879b-956d93393cbf-maas-api-tls\") pod \"maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp\" (UID: \"e1a979ed-3892-4eb5-879b-956d93393cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:44:35.357267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.357094    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lpqv\" (UniqueName: \"kubernetes.io/projected/e1a979ed-3892-4eb5-879b-956d93393cbf-kube-api-access-6lpqv\") pod \"maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp\" (UID: \"e1a979ed-3892-4eb5-879b-956d93393cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:44:35.357267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.357221    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a979ed-3892-4eb5-879b-956d93393cbf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp\" (UID: \"e1a979ed-3892-4eb5-879b-956d93393cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:44:35.357443 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.357421    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1a979ed-3892-4eb5-879b-956d93393cbf-metrics-certs\") pod \"maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp\" (UID: \"e1a979ed-3892-4eb5-879b-956d93393cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:44:35.378203 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.378164    4428 generic.go:356] "Generic (PLEG): container finished" podID="30f7fe62-925f-48d0-a15e-ed6c4eb56f2b" containerID="bf04cb843d2f5a6404f855500c246184f6fe4a4d1248ad41576714064e188ba6" exitCode=0
Oct 31 13:44:35.378410 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.378227    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv" event={"ID":"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b","Type":"ContainerDied","Data":"bf04cb843d2f5a6404f855500c246184f6fe4a4d1248ad41576714064e188ba6"}
Oct 31 13:44:35.447905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.446014    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:35.469925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.466981    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lpqv\" (UniqueName: \"kubernetes.io/projected/e1a979ed-3892-4eb5-879b-956d93393cbf-kube-api-access-6lpqv\") pod \"maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp\" (UID: \"e1a979ed-3892-4eb5-879b-956d93393cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:44:35.469925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.467072    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a979ed-3892-4eb5-879b-956d93393cbf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp\" (UID: \"e1a979ed-3892-4eb5-879b-956d93393cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:44:35.469925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.467272    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1a979ed-3892-4eb5-879b-956d93393cbf-metrics-certs\") pod \"maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp\" (UID: \"e1a979ed-3892-4eb5-879b-956d93393cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:44:35.469925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.469720    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a979ed-3892-4eb5-879b-956d93393cbf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp\" (UID: \"e1a979ed-3892-4eb5-879b-956d93393cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:44:35.470581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.470495    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1a979ed-3892-4eb5-879b-956d93393cbf-maas-api-tls\") pod \"maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp\" (UID: \"e1a979ed-3892-4eb5-879b-956d93393cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:44:35.473897 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:44:35.470749    4428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-serving-cert: secret "maas-api-e2e-inf-b-72c210-serving-cert" not found
Oct 31 13:44:35.473897 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:44:35.470821    4428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a979ed-3892-4eb5-879b-956d93393cbf-maas-api-tls podName:e1a979ed-3892-4eb5-879b-956d93393cbf nodeName:}" failed. No retries permitted until 2026-10-31 13:44:35.970801633 +0000 UTC m=+2473.369150254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e1a979ed-3892-4eb5-879b-956d93393cbf-maas-api-tls") pod "maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp" (UID: "e1a979ed-3892-4eb5-879b-956d93393cbf") : secret "maas-api-e2e-inf-b-72c210-serving-cert" not found
Oct 31 13:44:35.480560 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.480521    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1a979ed-3892-4eb5-879b-956d93393cbf-metrics-certs\") pod \"maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp\" (UID: \"e1a979ed-3892-4eb5-879b-956d93393cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:44:35.494967 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.490536    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lpqv\" (UniqueName: \"kubernetes.io/projected/e1a979ed-3892-4eb5-879b-956d93393cbf-kube-api-access-6lpqv\") pod \"maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp\" (UID: \"e1a979ed-3892-4eb5-879b-956d93393cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:44:35.605390 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.605301    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:35.772545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.772505    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:35.991897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:35.989369    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1a979ed-3892-4eb5-879b-956d93393cbf-maas-api-tls\") pod \"maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp\" (UID: \"e1a979ed-3892-4eb5-879b-956d93393cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:44:36.045894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:36.043507    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1a979ed-3892-4eb5-879b-956d93393cbf-maas-api-tls\") pod \"maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp\" (UID: \"e1a979ed-3892-4eb5-879b-956d93393cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:44:36.216083 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:36.216046    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:44:36.495908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:36.482774    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:36.548941 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:44:36.547385    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a979ed_3892_4eb5_879b_956d93393cbf.slice/crio-515ab74c478be8edb1ebf62fe9dcda5f7d825ef30846c865a459275fd4e1d3cc WatchSource:0}: Error finding container 515ab74c478be8edb1ebf62fe9dcda5f7d825ef30846c865a459275fd4e1d3cc: Status 404 returned error can't find the container with id 515ab74c478be8edb1ebf62fe9dcda5f7d825ef30846c865a459275fd4e1d3cc
Oct 31 13:44:36.549724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:36.549698    4428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:44:36.557180 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:36.557153    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"]
Oct 31 13:44:36.714553 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:36.714524    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"
Oct 31 13:44:36.754133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:36.754104    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-maas-api-openshift-ca\") pod \"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b\" (UID: \"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b\") "
Oct 31 13:44:36.754376 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:36.754215    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-kube-api-access-fdzcm\" (UniqueName: \"kubernetes.io/projected/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-kube-api-access-fdzcm\") pod \"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b\" (UID: \"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b\") "
Oct 31 13:44:36.754552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:36.754527    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-maas-api-openshift-ca" pod "30f7fe62-925f-48d0-a15e-ed6c4eb56f2b" (UID: "30f7fe62-925f-48d0-a15e-ed6c4eb56f2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:36.754656 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:36.754608    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-maas-api-xks-ca\") pod \"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b\" (UID: \"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b\") "
Oct 31 13:44:36.755406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:36.755380    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:36.757566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:36.757540    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-maas-api-xks-ca" pod "30f7fe62-925f-48d0-a15e-ed6c4eb56f2b" (UID: "30f7fe62-925f-48d0-a15e-ed6c4eb56f2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:44:36.757696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:36.757576    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-kube-api-access-fdzcm" pod "30f7fe62-925f-48d0-a15e-ed6c4eb56f2b" (UID: "30f7fe62-925f-48d0-a15e-ed6c4eb56f2b"). InnerVolumeSpecName "kube-api-access-fdzcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:36.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:36.857794    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:36.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:36.857823    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdzcm\" (UniqueName: \"kubernetes.io/projected/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b-kube-api-access-fdzcm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:37.333652 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.333615    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:37.467074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.467035    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"
Oct 31 13:44:37.467272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.467070    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv" event={"ID":"30f7fe62-925f-48d0-a15e-ed6c4eb56f2b","Type":"ContainerDied","Data":"9f503edd1f9beb5f08c517d606546b42c2bce7bc148d0443b605ef2ec734abbb"}
Oct 31 13:44:37.467272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.467123    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f503edd1f9beb5f08c517d606546b42c2bce7bc148d0443b605ef2ec734abbb"
Oct 31 13:44:37.469236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.469203    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp" event={"ID":"e1a979ed-3892-4eb5-879b-956d93393cbf","Type":"ContainerStarted","Data":"74309bb702e4963146ef98682671569f6a564cd34a7fb71b0a08c9097efbbdfc"}
Oct 31 13:44:37.469382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.469264    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp" event={"ID":"e1a979ed-3892-4eb5-879b-956d93393cbf","Type":"ContainerStarted","Data":"515ab74c478be8edb1ebf62fe9dcda5f7d825ef30846c865a459275fd4e1d3cc"}
Oct 31 13:44:37.469567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.469548    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:44:37.495211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.495157    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp" podStartSLOduration=1.973436652 podStartE2EDuration="2.495142961s" totalImagesPullingTime="521.706309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:44:37.492901105 +0000 UTC m=+2474.891249719" watchObservedRunningTime="2026-10-31 13:44:37.495142961 +0000 UTC m=+2474.893491603"
Oct 31 13:44:37.923682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.923653    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:44:37.981930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.981844    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-workload-certs\") pod \"c080a436-0943-4c33-938c-35b1ff2c1a11\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") "
Oct 31 13:44:37.982331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.982295    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c080a436-0943-4c33-938c-35b1ff2c1a11-kube-api-access-x67c2\" (UniqueName: \"kubernetes.io/projected/c080a436-0943-4c33-938c-35b1ff2c1a11-kube-api-access-x67c2\") pod \"c080a436-0943-4c33-938c-35b1ff2c1a11\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") "
Oct 31 13:44:37.982566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.982360    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-podinfo\") pod \"c080a436-0943-4c33-938c-35b1ff2c1a11\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") "
Oct 31 13:44:37.982566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.982445    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-workload-certs" pod "c080a436-0943-4c33-938c-35b1ff2c1a11" (UID: "c080a436-0943-4c33-938c-35b1ff2c1a11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:37.982566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.982520    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-data\") pod \"c080a436-0943-4c33-938c-35b1ff2c1a11\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") "
Oct 31 13:44:37.982851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.982619    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c080a436-0943-4c33-938c-35b1ff2c1a11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c080a436-0943-4c33-938c-35b1ff2c1a11-istiod-ca-cert\") pod \"c080a436-0943-4c33-938c-35b1ff2c1a11\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") "
Oct 31 13:44:37.982851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.982674    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-envoy\") pod \"c080a436-0943-4c33-938c-35b1ff2c1a11\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") "
Oct 31 13:44:37.982851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.982734    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-credential-socket\") pod \"c080a436-0943-4c33-938c-35b1ff2c1a11\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") "
Oct 31 13:44:37.982851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.982781    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-token\" (UniqueName: \"kubernetes.io/projected/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-token\") pod \"c080a436-0943-4c33-938c-35b1ff2c1a11\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") "
Oct 31 13:44:37.983289 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.983116    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-workload-socket\") pod \"c080a436-0943-4c33-938c-35b1ff2c1a11\" (UID: \"c080a436-0943-4c33-938c-35b1ff2c1a11\") "
Oct 31 13:44:37.984555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.983724    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-data" pod "c080a436-0943-4c33-938c-35b1ff2c1a11" (UID: "c080a436-0943-4c33-938c-35b1ff2c1a11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:37.984555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.984081    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-credential-socket" pod "c080a436-0943-4c33-938c-35b1ff2c1a11" (UID: "c080a436-0943-4c33-938c-35b1ff2c1a11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:37.984555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.984124    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-workload-socket" pod "c080a436-0943-4c33-938c-35b1ff2c1a11" (UID: "c080a436-0943-4c33-938c-35b1ff2c1a11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:37.985053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.984967    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:37.985053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.984995    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:37.985053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.985015    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:37.985053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.985030    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:37.985274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.985081    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c080a436-0943-4c33-938c-35b1ff2c1a11-istiod-ca-cert" pod "c080a436-0943-4c33-938c-35b1ff2c1a11" (UID: "c080a436-0943-4c33-938c-35b1ff2c1a11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:37.987994 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.987827    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-token" pod "c080a436-0943-4c33-938c-35b1ff2c1a11" (UID: "c080a436-0943-4c33-938c-35b1ff2c1a11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:37.987994 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.987951    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-envoy" pod "c080a436-0943-4c33-938c-35b1ff2c1a11" (UID: "c080a436-0943-4c33-938c-35b1ff2c1a11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:37.993698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.993666    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-podinfo" pod "c080a436-0943-4c33-938c-35b1ff2c1a11" (UID: "c080a436-0943-4c33-938c-35b1ff2c1a11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:44:37.994058 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:37.994028    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c080a436-0943-4c33-938c-35b1ff2c1a11-kube-api-access-x67c2" pod "c080a436-0943-4c33-938c-35b1ff2c1a11" (UID: "c080a436-0943-4c33-938c-35b1ff2c1a11"). InnerVolumeSpecName "kube-api-access-x67c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:38.086995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.086906    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x67c2\" (UniqueName: \"kubernetes.io/projected/c080a436-0943-4c33-938c-35b1ff2c1a11-kube-api-access-x67c2\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:38.086995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.086934    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:38.086995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.086946    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c080a436-0943-4c33-938c-35b1ff2c1a11-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:38.086995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.086955    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:38.086995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.086964    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c080a436-0943-4c33-938c-35b1ff2c1a11-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:38.233897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.233166    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:38.482229 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.482197    4428 generic.go:356] "Generic (PLEG): container finished" podID="c080a436-0943-4c33-938c-35b1ff2c1a11" containerID="c6699006304b01ee77cdd77437b3100340400b010222ebb90ddbb9343f322c9b" exitCode=0
Oct 31 13:44:38.482444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.482255    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"
Oct 31 13:44:38.482444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.482287    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x" event={"ID":"c080a436-0943-4c33-938c-35b1ff2c1a11","Type":"ContainerDied","Data":"c6699006304b01ee77cdd77437b3100340400b010222ebb90ddbb9343f322c9b"}
Oct 31 13:44:38.482444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.482350    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x" event={"ID":"c080a436-0943-4c33-938c-35b1ff2c1a11","Type":"ContainerDied","Data":"9b92fb041dd2cdd9aa7afbff08cfde337fbb0c3d5c23124ce5b07bf80d64cb05"}
Oct 31 13:44:38.482444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.482374    4428 scope.go:122] "RemoveContainer" containerID="c6699006304b01ee77cdd77437b3100340400b010222ebb90ddbb9343f322c9b"
Oct 31 13:44:38.510272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.510222    4428 scope.go:122] "RemoveContainer" containerID="d6ebff4bee6e489501f3c38883f9f91780938f8f20ade5c84b5c7d5ac26085a8"
Oct 31 13:44:38.515634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.515607    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"]
Oct 31 13:44:38.523423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.523402    4428 scope.go:122] "RemoveContainer" containerID="c6699006304b01ee77cdd77437b3100340400b010222ebb90ddbb9343f322c9b"
Oct 31 13:44:38.523842 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:44:38.523817    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6699006304b01ee77cdd77437b3100340400b010222ebb90ddbb9343f322c9b\": container with ID starting with c6699006304b01ee77cdd77437b3100340400b010222ebb90ddbb9343f322c9b not found: ID does not exist" containerID="c6699006304b01ee77cdd77437b3100340400b010222ebb90ddbb9343f322c9b"
Oct 31 13:44:38.524008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.523853    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6699006304b01ee77cdd77437b3100340400b010222ebb90ddbb9343f322c9b"} err="failed to get container status \"c6699006304b01ee77cdd77437b3100340400b010222ebb90ddbb9343f322c9b\": rpc error: code = NotFound desc = could not find container \"c6699006304b01ee77cdd77437b3100340400b010222ebb90ddbb9343f322c9b\": container with ID starting with c6699006304b01ee77cdd77437b3100340400b010222ebb90ddbb9343f322c9b not found: ID does not exist"
Oct 31 13:44:38.524008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.523897    4428 scope.go:122] "RemoveContainer" containerID="d6ebff4bee6e489501f3c38883f9f91780938f8f20ade5c84b5c7d5ac26085a8"
Oct 31 13:44:38.524273 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:44:38.524246    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6ebff4bee6e489501f3c38883f9f91780938f8f20ade5c84b5c7d5ac26085a8\": container with ID starting with d6ebff4bee6e489501f3c38883f9f91780938f8f20ade5c84b5c7d5ac26085a8 not found: ID does not exist" containerID="d6ebff4bee6e489501f3c38883f9f91780938f8f20ade5c84b5c7d5ac26085a8"
Oct 31 13:44:38.524359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.524281    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6ebff4bee6e489501f3c38883f9f91780938f8f20ade5c84b5c7d5ac26085a8"} err="failed to get container status \"d6ebff4bee6e489501f3c38883f9f91780938f8f20ade5c84b5c7d5ac26085a8\": rpc error: code = NotFound desc = could not find container \"d6ebff4bee6e489501f3c38883f9f91780938f8f20ade5c84b5c7d5ac26085a8\": container with ID starting with d6ebff4bee6e489501f3c38883f9f91780938f8f20ade5c84b5c7d5ac26085a8 not found: ID does not exist"
Oct 31 13:44:38.528207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.528185    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-td25x"]
Oct 31 13:44:38.759483 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:38.759397    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c080a436-0943-4c33-938c-35b1ff2c1a11" path="/var/lib/kubelet/pods/c080a436-0943-4c33-938c-35b1ff2c1a11/volumes"
Oct 31 13:44:39.026959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:39.020682    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:39.905896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:39.901088    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:40.704343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:40.700504    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:41.632682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:41.632645    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:42.335490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:42.335457    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"]
Oct 31 13:44:42.336090 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:42.335713    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz" podUID="aa7b17e1-a025-428d-ad57-a52013e967fc" containerName="istio-proxy" containerID="cri-o://529346514e652c225d3955628b66c6bf35147b0b7be6ef443a998906673d46fd" gracePeriod=30
Oct 31 13:44:42.483211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:42.483170    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:42.993087 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:42.993053    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"]
Oct 31 13:44:43.005410 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:43.005373    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"]
Oct 31 13:44:43.126682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:43.126634    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:43.314960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:43.309123    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:43.508309 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:43.505489    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:43.525891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:43.512063    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:44:44.403925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:44.403894    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:45.235896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:45.217638    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:46.101882 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:46.101826    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:46.116914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:46.116862    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"]
Oct 31 13:44:46.117112 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:46.117089    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb" podUID="fb96175b-4567-4467-a2fa-1aacfeffa985" containerName="istio-proxy" containerID="cri-o://be4d146af5e6902055d82ab6c3d5e13ffc97820438b61536634b0dbc8f7bde20" gracePeriod=30
Oct 31 13:44:46.942515 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:46.942467    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:47.427292 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.427260    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:47.599905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.599835    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:47.649705 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.649673    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:44:47.655407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.655378    4428 generic.go:356] "Generic (PLEG): container finished" podID="aa7b17e1-a025-428d-ad57-a52013e967fc" containerID="529346514e652c225d3955628b66c6bf35147b0b7be6ef443a998906673d46fd" exitCode=0
Oct 31 13:44:47.655561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.655421    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"
Oct 31 13:44:47.655561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.655454    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz" event={"ID":"aa7b17e1-a025-428d-ad57-a52013e967fc","Type":"ContainerDied","Data":"529346514e652c225d3955628b66c6bf35147b0b7be6ef443a998906673d46fd"}
Oct 31 13:44:47.655561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.655508    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz" event={"ID":"aa7b17e1-a025-428d-ad57-a52013e967fc","Type":"ContainerDied","Data":"f73ce46c6ec44dfea14db74175f5c427ec55b7a2568ad01437f6dec0c96a6ba9"}
Oct 31 13:44:47.655561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.655531    4428 scope.go:122] "RemoveContainer" containerID="529346514e652c225d3955628b66c6bf35147b0b7be6ef443a998906673d46fd"
Oct 31 13:44:47.667052 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.667034    4428 scope.go:122] "RemoveContainer" containerID="529346514e652c225d3955628b66c6bf35147b0b7be6ef443a998906673d46fd"
Oct 31 13:44:47.667373 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:44:47.667348    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529346514e652c225d3955628b66c6bf35147b0b7be6ef443a998906673d46fd\": container with ID starting with 529346514e652c225d3955628b66c6bf35147b0b7be6ef443a998906673d46fd not found: ID does not exist" containerID="529346514e652c225d3955628b66c6bf35147b0b7be6ef443a998906673d46fd"
Oct 31 13:44:47.667454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.667387    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529346514e652c225d3955628b66c6bf35147b0b7be6ef443a998906673d46fd"} err="failed to get container status \"529346514e652c225d3955628b66c6bf35147b0b7be6ef443a998906673d46fd\": rpc error: code = NotFound desc = could not find container \"529346514e652c225d3955628b66c6bf35147b0b7be6ef443a998906673d46fd\": container with ID starting with 529346514e652c225d3955628b66c6bf35147b0b7be6ef443a998906673d46fd not found: ID does not exist"
Oct 31 13:44:47.758085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.758048    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-token\" (UniqueName: \"kubernetes.io/projected/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-token\") pod \"aa7b17e1-a025-428d-ad57-a52013e967fc\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") "
Oct 31 13:44:47.758270 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.758115    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-workload-certs\") pod \"aa7b17e1-a025-428d-ad57-a52013e967fc\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") "
Oct 31 13:44:47.758270 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.758248    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-workload-socket\") pod \"aa7b17e1-a025-428d-ad57-a52013e967fc\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") "
Oct 31 13:44:47.758417 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.758287    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa7b17e1-a025-428d-ad57-a52013e967fc-kube-api-access-jmsss\" (UniqueName: \"kubernetes.io/projected/aa7b17e1-a025-428d-ad57-a52013e967fc-kube-api-access-jmsss\") pod \"aa7b17e1-a025-428d-ad57-a52013e967fc\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") "
Oct 31 13:44:47.758417 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.758332    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-podinfo\") pod \"aa7b17e1-a025-428d-ad57-a52013e967fc\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") "
Oct 31 13:44:47.758523 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.758454    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa7b17e1-a025-428d-ad57-a52013e967fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7b17e1-a025-428d-ad57-a52013e967fc-istiod-ca-cert\") pod \"aa7b17e1-a025-428d-ad57-a52013e967fc\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") "
Oct 31 13:44:47.758523 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.758492    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-workload-certs" pod "aa7b17e1-a025-428d-ad57-a52013e967fc" (UID: "aa7b17e1-a025-428d-ad57-a52013e967fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:47.758631 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.758539    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-workload-socket" pod "aa7b17e1-a025-428d-ad57-a52013e967fc" (UID: "aa7b17e1-a025-428d-ad57-a52013e967fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:47.758631 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.758567    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-credential-socket\") pod \"aa7b17e1-a025-428d-ad57-a52013e967fc\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") "
Oct 31 13:44:47.758731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.758669    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-data\") pod \"aa7b17e1-a025-428d-ad57-a52013e967fc\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") "
Oct 31 13:44:47.758731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.758701    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-envoy\") pod \"aa7b17e1-a025-428d-ad57-a52013e967fc\" (UID: \"aa7b17e1-a025-428d-ad57-a52013e967fc\") "
Oct 31 13:44:47.758963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.758934    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7b17e1-a025-428d-ad57-a52013e967fc-istiod-ca-cert" pod "aa7b17e1-a025-428d-ad57-a52013e967fc" (UID: "aa7b17e1-a025-428d-ad57-a52013e967fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:47.759206 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.759072    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-credential-socket" pod "aa7b17e1-a025-428d-ad57-a52013e967fc" (UID: "aa7b17e1-a025-428d-ad57-a52013e967fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:47.759356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.759110    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-data" pod "aa7b17e1-a025-428d-ad57-a52013e967fc" (UID: "aa7b17e1-a025-428d-ad57-a52013e967fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:47.759792 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.759721    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:47.759792 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.759759    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa7b17e1-a025-428d-ad57-a52013e967fc-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:47.759792 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.759816    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:47.759792 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.759832    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:47.759792 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.759847    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:47.761555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.761502    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-podinfo" pod "aa7b17e1-a025-428d-ad57-a52013e967fc" (UID: "aa7b17e1-a025-428d-ad57-a52013e967fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:44:47.761774 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.761735    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-envoy" pod "aa7b17e1-a025-428d-ad57-a52013e967fc" (UID: "aa7b17e1-a025-428d-ad57-a52013e967fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:47.762589 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.762568    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-token" pod "aa7b17e1-a025-428d-ad57-a52013e967fc" (UID: "aa7b17e1-a025-428d-ad57-a52013e967fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:47.763459 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.763433    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7b17e1-a025-428d-ad57-a52013e967fc-kube-api-access-jmsss" pod "aa7b17e1-a025-428d-ad57-a52013e967fc" (UID: "aa7b17e1-a025-428d-ad57-a52013e967fc"). InnerVolumeSpecName "kube-api-access-jmsss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:47.861787 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.861706    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmsss\" (UniqueName: \"kubernetes.io/projected/aa7b17e1-a025-428d-ad57-a52013e967fc-kube-api-access-jmsss\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:47.861787 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.861738    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:47.861787 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.861749    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:47.861787 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.861758    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7b17e1-a025-428d-ad57-a52013e967fc-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:47.981902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.981002    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:47.999627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:47.995632    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p" podUID="9cf05b3f-e7f7-4b17-b06c-c015ae338bad" containerName="payload-processing" containerID="cri-o://f4af763bfc4b28c4c6b9ad53de8620dee77c08acb22ba5930134797bed7dc5e6" gracePeriod=25
Oct 31 13:44:48.008263 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:48.006451    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s" podUID="dd12e2cb-e569-446e-8cbf-d6cd4779c833" containerName="payload-pre-processing" containerID="cri-o://765142255aff4fff167f6035389dd098a29fed8cb1425591e6107036ff29c1ee" gracePeriod=25
Oct 31 13:44:48.017433 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:48.017393    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"]
Oct 31 13:44:48.026761 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:48.026726    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-dghcz"]
Oct 31 13:44:48.743697 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:44:48.743656    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:44:48.759043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:48.759005    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa7b17e1-a025-428d-ad57-a52013e967fc" path="/var/lib/kubelet/pods/aa7b17e1-a025-428d-ad57-a52013e967fc/volumes"
Oct 31 13:44:48.852616 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:48.851393    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:49.836826 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:49.836569    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:50.719894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:50.700650    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:51.418838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.418810    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:44:51.479931 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.479861    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-token\" (UniqueName: \"kubernetes.io/projected/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-token\") pod \"fb96175b-4567-4467-a2fa-1aacfeffa985\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") "
Oct 31 13:44:51.480160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.479959    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-workload-socket\") pod \"fb96175b-4567-4467-a2fa-1aacfeffa985\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") "
Oct 31 13:44:51.480160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.480032    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-data\") pod \"fb96175b-4567-4467-a2fa-1aacfeffa985\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") "
Oct 31 13:44:51.480160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.480062    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-podinfo\") pod \"fb96175b-4567-4467-a2fa-1aacfeffa985\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") "
Oct 31 13:44:51.480160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.480100    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb96175b-4567-4467-a2fa-1aacfeffa985-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb96175b-4567-4467-a2fa-1aacfeffa985-istiod-ca-cert\") pod \"fb96175b-4567-4467-a2fa-1aacfeffa985\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") "
Oct 31 13:44:51.480412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.480170    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-workload-certs\") pod \"fb96175b-4567-4467-a2fa-1aacfeffa985\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") "
Oct 31 13:44:51.480412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.480261    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-credential-socket\") pod \"fb96175b-4567-4467-a2fa-1aacfeffa985\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") "
Oct 31 13:44:51.480412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.480346    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-envoy\") pod \"fb96175b-4567-4467-a2fa-1aacfeffa985\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") "
Oct 31 13:44:51.480412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.480386    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb96175b-4567-4467-a2fa-1aacfeffa985-kube-api-access-9bplr\" (UniqueName: \"kubernetes.io/projected/fb96175b-4567-4467-a2fa-1aacfeffa985-kube-api-access-9bplr\") pod \"fb96175b-4567-4467-a2fa-1aacfeffa985\" (UID: \"fb96175b-4567-4467-a2fa-1aacfeffa985\") "
Oct 31 13:44:51.481547 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.481412    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-workload-certs" pod "fb96175b-4567-4467-a2fa-1aacfeffa985" (UID: "fb96175b-4567-4467-a2fa-1aacfeffa985"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:51.481816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.481732    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-workload-socket" pod "fb96175b-4567-4467-a2fa-1aacfeffa985" (UID: "fb96175b-4567-4467-a2fa-1aacfeffa985"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:51.482100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.482009    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb96175b-4567-4467-a2fa-1aacfeffa985-istiod-ca-cert" pod "fb96175b-4567-4467-a2fa-1aacfeffa985" (UID: "fb96175b-4567-4467-a2fa-1aacfeffa985"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:44:51.482208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.482117    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-data" pod "fb96175b-4567-4467-a2fa-1aacfeffa985" (UID: "fb96175b-4567-4467-a2fa-1aacfeffa985"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:51.483061 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.482806    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-credential-socket" pod "fb96175b-4567-4467-a2fa-1aacfeffa985" (UID: "fb96175b-4567-4467-a2fa-1aacfeffa985"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:51.486279 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.486248    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-podinfo" pod "fb96175b-4567-4467-a2fa-1aacfeffa985" (UID: "fb96175b-4567-4467-a2fa-1aacfeffa985"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:44:51.486767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.486591    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb96175b-4567-4467-a2fa-1aacfeffa985-kube-api-access-9bplr" pod "fb96175b-4567-4467-a2fa-1aacfeffa985" (UID: "fb96175b-4567-4467-a2fa-1aacfeffa985"). InnerVolumeSpecName "kube-api-access-9bplr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:51.486767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.486650    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-token" pod "fb96175b-4567-4467-a2fa-1aacfeffa985" (UID: "fb96175b-4567-4467-a2fa-1aacfeffa985"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:44:51.486767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.486742    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-envoy" pod "fb96175b-4567-4467-a2fa-1aacfeffa985" (UID: "fb96175b-4567-4467-a2fa-1aacfeffa985"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:44:51.497306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.497270    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:51.637147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.602542    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bplr\" (UniqueName: \"kubernetes.io/projected/fb96175b-4567-4467-a2fa-1aacfeffa985-kube-api-access-9bplr\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:51.637147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.602589    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:51.637147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.602605    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:51.637147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.602619    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:51.637147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.602635    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:51.637147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.602649    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb96175b-4567-4467-a2fa-1aacfeffa985-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:51.637147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.602683    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:51.637147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.602698    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:51.637147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.602712    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb96175b-4567-4467-a2fa-1aacfeffa985-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:44:51.765415 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.765375    4428 generic.go:356] "Generic (PLEG): container finished" podID="fb96175b-4567-4467-a2fa-1aacfeffa985" containerID="be4d146af5e6902055d82ab6c3d5e13ffc97820438b61536634b0dbc8f7bde20" exitCode=0
Oct 31 13:44:51.765607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.765429    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"
Oct 31 13:44:51.765607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.765436    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb" event={"ID":"fb96175b-4567-4467-a2fa-1aacfeffa985","Type":"ContainerDied","Data":"be4d146af5e6902055d82ab6c3d5e13ffc97820438b61536634b0dbc8f7bde20"}
Oct 31 13:44:51.765607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.765489    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb" event={"ID":"fb96175b-4567-4467-a2fa-1aacfeffa985","Type":"ContainerDied","Data":"f6e14f2219576301478294e84ccd83311ad1afb20e367ff45b7e08d32a34db64"}
Oct 31 13:44:51.765607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.765520    4428 scope.go:122] "RemoveContainer" containerID="be4d146af5e6902055d82ab6c3d5e13ffc97820438b61536634b0dbc8f7bde20"
Oct 31 13:44:51.778511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.778488    4428 scope.go:122] "RemoveContainer" containerID="be4d146af5e6902055d82ab6c3d5e13ffc97820438b61536634b0dbc8f7bde20"
Oct 31 13:44:51.778916 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:44:51.778895    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4d146af5e6902055d82ab6c3d5e13ffc97820438b61536634b0dbc8f7bde20\": container with ID starting with be4d146af5e6902055d82ab6c3d5e13ffc97820438b61536634b0dbc8f7bde20 not found: ID does not exist" containerID="be4d146af5e6902055d82ab6c3d5e13ffc97820438b61536634b0dbc8f7bde20"
Oct 31 13:44:51.778984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.778926    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4d146af5e6902055d82ab6c3d5e13ffc97820438b61536634b0dbc8f7bde20"} err="failed to get container status \"be4d146af5e6902055d82ab6c3d5e13ffc97820438b61536634b0dbc8f7bde20\": rpc error: code = NotFound desc = could not find container \"be4d146af5e6902055d82ab6c3d5e13ffc97820438b61536634b0dbc8f7bde20\": container with ID starting with be4d146af5e6902055d82ab6c3d5e13ffc97820438b61536634b0dbc8f7bde20 not found: ID does not exist"
Oct 31 13:44:51.796264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.796236    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"]
Oct 31 13:44:51.801188 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:51.801165    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-mlgfb"]
Oct 31 13:44:52.064474 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:52.064333    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:52.264218 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:52.264168    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:52.758294 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:52.758260    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb96175b-4567-4467-a2fa-1aacfeffa985" path="/var/lib/kubelet/pods/fb96175b-4567-4467-a2fa-1aacfeffa985/volumes"
Oct 31 13:44:53.137887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:53.137770    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:54.021433 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:54.021397    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:55.004540 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:55.004491    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:55.871246 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:55.871207    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:56.075130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:56.075076    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:56.176965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:56.173053    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p" podUID="9cf05b3f-e7f7-4b17-b06c-c015ae338bad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:44:56.177513 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:56.177477    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s" podUID="dd12e2cb-e569-446e-8cbf-d6cd4779c833" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:44:56.893424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:56.893388    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:57.983208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:57.983170    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:58.902367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:58.902321    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:59.197956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:59.186500    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:59.492091 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:59.492052    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"]
Oct 31 13:44:59.498361 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:59.498317    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d4b52a-lshxv"]
Oct 31 13:44:59.528939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:59.528903    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:44:59.812107 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:44:59.807847    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:00.071962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.071789    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:00.160379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.160340    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76"]
Oct 31 13:45:00.161655 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.161629    4428 cpu_manager.go:414] "containerMap: removing container" podUID="fb96175b-4567-4467-a2fa-1aacfeffa985" containerName="istio-proxy"
Oct 31 13:45:00.161655 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.161655    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb96175b-4567-4467-a2fa-1aacfeffa985" containerName="istio-proxy"
Oct 31 13:45:00.161912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.161672    4428 cpu_manager.go:414] "containerMap: removing container" podUID="30f7fe62-925f-48d0-a15e-ed6c4eb56f2b" containerName="revoke-keys"
Oct 31 13:45:00.161912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.161681    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f7fe62-925f-48d0-a15e-ed6c4eb56f2b" containerName="revoke-keys"
Oct 31 13:45:00.161912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.161700    4428 cpu_manager.go:414] "containerMap: removing container" podUID="aa7b17e1-a025-428d-ad57-a52013e967fc" containerName="istio-proxy"
Oct 31 13:45:00.161912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.161708    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa7b17e1-a025-428d-ad57-a52013e967fc" containerName="istio-proxy"
Oct 31 13:45:00.161912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.161727    4428 cpu_manager.go:414] "containerMap: removing container" podUID="c080a436-0943-4c33-938c-35b1ff2c1a11" containerName="istio-proxy"
Oct 31 13:45:00.161912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.161733    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c080a436-0943-4c33-938c-35b1ff2c1a11" containerName="istio-proxy"
Oct 31 13:45:00.162196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.161969    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c080a436-0943-4c33-938c-35b1ff2c1a11" containerName="istio-proxy"
Oct 31 13:45:00.162196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.161988    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb96175b-4567-4467-a2fa-1aacfeffa985" containerName="istio-proxy"
Oct 31 13:45:00.162196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.162000    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f7fe62-925f-48d0-a15e-ed6c4eb56f2b" containerName="revoke-keys"
Oct 31 13:45:00.162196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.162010    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa7b17e1-a025-428d-ad57-a52013e967fc" containerName="istio-proxy"
Oct 31 13:45:00.169041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.168855    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp"]
Oct 31 13:45:00.170413 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.170388    4428 cpu_manager.go:414] "containerMap: removing container" podUID="c080a436-0943-4c33-938c-35b1ff2c1a11" containerName="istio-proxy"
Oct 31 13:45:00.170567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.170552    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c080a436-0943-4c33-938c-35b1ff2c1a11" containerName="istio-proxy"
Oct 31 13:45:00.171044 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.171026    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c080a436-0943-4c33-938c-35b1ff2c1a11" containerName="istio-proxy"
Oct 31 13:45:00.171494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.171476    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76"
Oct 31 13:45:00.201066 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.199811    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:45:00.216291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.203990    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76"]
Oct 31 13:45:00.216291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.204048    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp"]
Oct 31 13:45:00.216291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.204075    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8"]
Oct 31 13:45:00.216291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.208338    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp"
Oct 31 13:45:00.216291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.209103    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w9ps\" (UniqueName: \"kubernetes.io/projected/d507266c-1d74-4d2c-9af8-c9990fe98174-kube-api-access-2w9ps\") pod \"maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76\" (UID: \"d507266c-1d74-4d2c-9af8-c9990fe98174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76"
Oct 31 13:45:00.216291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.209295    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d507266c-1d74-4d2c-9af8-c9990fe98174-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76\" (UID: \"d507266c-1d74-4d2c-9af8-c9990fe98174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76"
Oct 31 13:45:00.216291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.211097    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8"
Oct 31 13:45:00.221213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.220546    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp"]
Oct 31 13:45:00.232726 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.232578    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp"
Oct 31 13:45:00.250561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.250529    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp"]
Oct 31 13:45:00.260060 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.259759    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8"]
Oct 31 13:45:00.276324 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.276294    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7"]
Oct 31 13:45:00.283618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.283591    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7"
Oct 31 13:45:00.284970 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.284946    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs"]
Oct 31 13:45:00.307483 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.307456    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs"
Oct 31 13:45:00.313154 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.313116    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w9ps\" (UniqueName: \"kubernetes.io/projected/d507266c-1d74-4d2c-9af8-c9990fe98174-kube-api-access-2w9ps\") pod \"maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76\" (UID: \"d507266c-1d74-4d2c-9af8-c9990fe98174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76"
Oct 31 13:45:00.313348 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.313202    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e615817e-f1b7-485a-b4d4-b63b54b6db48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7\" (UID: \"e615817e-f1b7-485a-b4d4-b63b54b6db48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7"
Oct 31 13:45:00.313348 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.313258    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9daa213c-e1ea-473d-8ed6-a2bc15630b58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp\" (UID: \"9daa213c-e1ea-473d-8ed6-a2bc15630b58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp"
Oct 31 13:45:00.313348 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.313320    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8tkm\" (UniqueName: \"kubernetes.io/projected/680bd885-b3c7-42f5-a3a8-4f2950d0b462-kube-api-access-q8tkm\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp\" (UID: \"680bd885-b3c7-42f5-a3a8-4f2950d0b462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp"
Oct 31 13:45:00.313588 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.313372    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d507266c-1d74-4d2c-9af8-c9990fe98174-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76\" (UID: \"d507266c-1d74-4d2c-9af8-c9990fe98174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76"
Oct 31 13:45:00.313588 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.313410    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb5549e-6008-46a8-8e55-3b37cb62cd60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8\" (UID: \"6bb5549e-6008-46a8-8e55-3b37cb62cd60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8"
Oct 31 13:45:00.313588 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.313453    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw6hh\" (UniqueName: \"kubernetes.io/projected/e615817e-f1b7-485a-b4d4-b63b54b6db48-kube-api-access-qw6hh\") pod \"maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7\" (UID: \"e615817e-f1b7-485a-b4d4-b63b54b6db48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7"
Oct 31 13:45:00.313588 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.313565    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6bdt\" (UniqueName: \"kubernetes.io/projected/9daa213c-e1ea-473d-8ed6-a2bc15630b58-kube-api-access-m6bdt\") pod \"maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp\" (UID: \"9daa213c-e1ea-473d-8ed6-a2bc15630b58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp"
Oct 31 13:45:00.314019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.313654    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sfnj\" (UniqueName: \"kubernetes.io/projected/6bb5549e-6008-46a8-8e55-3b37cb62cd60-kube-api-access-8sfnj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8\" (UID: \"6bb5549e-6008-46a8-8e55-3b37cb62cd60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8"
Oct 31 13:45:00.314019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.313726    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/680bd885-b3c7-42f5-a3a8-4f2950d0b462-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp\" (UID: \"680bd885-b3c7-42f5-a3a8-4f2950d0b462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp"
Oct 31 13:45:00.316346 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.316323    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7"]
Oct 31 13:45:00.318314 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.318101    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d507266c-1d74-4d2c-9af8-c9990fe98174-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76\" (UID: \"d507266c-1d74-4d2c-9af8-c9990fe98174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76"
Oct 31 13:45:00.329697 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.329670    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs"]
Oct 31 13:45:00.331072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.331053    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w9ps\" (UniqueName: \"kubernetes.io/projected/d507266c-1d74-4d2c-9af8-c9990fe98174-kube-api-access-2w9ps\") pod \"maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76\" (UID: \"d507266c-1d74-4d2c-9af8-c9990fe98174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76"
Oct 31 13:45:00.339774 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.335914    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4"]
Oct 31 13:45:00.342093 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.342057    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4"
Oct 31 13:45:00.355467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.355440    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4"]
Oct 31 13:45:00.362108 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.362081    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8"]
Oct 31 13:45:00.373611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.373583    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67"]
Oct 31 13:45:00.374462 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.373983    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8"
Oct 31 13:45:00.393983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.393911    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8"]
Oct 31 13:45:00.395360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.394307    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67"
Oct 31 13:45:00.415020 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.414969    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw"]
Oct 31 13:45:00.417423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.417383    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb5549e-6008-46a8-8e55-3b37cb62cd60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8\" (UID: \"6bb5549e-6008-46a8-8e55-3b37cb62cd60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8"
Oct 31 13:45:00.417597 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.417454    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw6hh\" (UniqueName: \"kubernetes.io/projected/e615817e-f1b7-485a-b4d4-b63b54b6db48-kube-api-access-qw6hh\") pod \"maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7\" (UID: \"e615817e-f1b7-485a-b4d4-b63b54b6db48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7"
Oct 31 13:45:00.417597 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.417559    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6498d\" (UniqueName: \"kubernetes.io/projected/77411480-49b1-49e5-90f0-1fc4416288da-kube-api-access-6498d\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs\" (UID: \"77411480-49b1-49e5-90f0-1fc4416288da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs"
Oct 31 13:45:00.417775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.417602    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6bdt\" (UniqueName: \"kubernetes.io/projected/9daa213c-e1ea-473d-8ed6-a2bc15630b58-kube-api-access-m6bdt\") pod \"maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp\" (UID: \"9daa213c-e1ea-473d-8ed6-a2bc15630b58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp"
Oct 31 13:45:00.417775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.417636    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9485106e-58f9-4b42-ae34-0136f6345af2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4\" (UID: \"9485106e-58f9-4b42-ae34-0136f6345af2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4"
Oct 31 13:45:00.417775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.417711    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sfnj\" (UniqueName: \"kubernetes.io/projected/6bb5549e-6008-46a8-8e55-3b37cb62cd60-kube-api-access-8sfnj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8\" (UID: \"6bb5549e-6008-46a8-8e55-3b37cb62cd60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8"
Oct 31 13:45:00.417775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.417766    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fe9f57-eaa6-49c1-9a63-f37485554a23-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67\" (UID: \"61fe9f57-eaa6-49c1-9a63-f37485554a23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67"
Oct 31 13:45:00.418047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.417796    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcghq\" (UniqueName: \"kubernetes.io/projected/61fe9f57-eaa6-49c1-9a63-f37485554a23-kube-api-access-zcghq\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67\" (UID: \"61fe9f57-eaa6-49c1-9a63-f37485554a23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67"
Oct 31 13:45:00.418047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.417830    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/680bd885-b3c7-42f5-a3a8-4f2950d0b462-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp\" (UID: \"680bd885-b3c7-42f5-a3a8-4f2950d0b462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp"
Oct 31 13:45:00.418047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.417889    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmvrc\" (UniqueName: \"kubernetes.io/projected/9485106e-58f9-4b42-ae34-0136f6345af2-kube-api-access-bmvrc\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4\" (UID: \"9485106e-58f9-4b42-ae34-0136f6345af2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4"
Oct 31 13:45:00.418047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.417971    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d9f9bc-22ec-4943-8a8c-9ed7abe7af98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8\" (UID: \"55d9f9bc-22ec-4943-8a8c-9ed7abe7af98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8"
Oct 31 13:45:00.418047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.418020    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85ssh\" (UniqueName: \"kubernetes.io/projected/55d9f9bc-22ec-4943-8a8c-9ed7abe7af98-kube-api-access-85ssh\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8\" (UID: \"55d9f9bc-22ec-4943-8a8c-9ed7abe7af98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8"
Oct 31 13:45:00.418356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.418170    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77411480-49b1-49e5-90f0-1fc4416288da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs\" (UID: \"77411480-49b1-49e5-90f0-1fc4416288da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs"
Oct 31 13:45:00.418356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.418204    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e615817e-f1b7-485a-b4d4-b63b54b6db48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7\" (UID: \"e615817e-f1b7-485a-b4d4-b63b54b6db48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7"
Oct 31 13:45:00.418356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.418262    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9daa213c-e1ea-473d-8ed6-a2bc15630b58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp\" (UID: \"9daa213c-e1ea-473d-8ed6-a2bc15630b58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp"
Oct 31 13:45:00.418356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.418336    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8tkm\" (UniqueName: \"kubernetes.io/projected/680bd885-b3c7-42f5-a3a8-4f2950d0b462-kube-api-access-q8tkm\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp\" (UID: \"680bd885-b3c7-42f5-a3a8-4f2950d0b462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp"
Oct 31 13:45:00.420354 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.419444    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb5549e-6008-46a8-8e55-3b37cb62cd60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8\" (UID: \"6bb5549e-6008-46a8-8e55-3b37cb62cd60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8"
Oct 31 13:45:00.420354 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.420162    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/680bd885-b3c7-42f5-a3a8-4f2950d0b462-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp\" (UID: \"680bd885-b3c7-42f5-a3a8-4f2950d0b462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp"
Oct 31 13:45:00.421212 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.420322    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e615817e-f1b7-485a-b4d4-b63b54b6db48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7\" (UID: \"e615817e-f1b7-485a-b4d4-b63b54b6db48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7"
Oct 31 13:45:00.421212 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.421204    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9daa213c-e1ea-473d-8ed6-a2bc15630b58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp\" (UID: \"9daa213c-e1ea-473d-8ed6-a2bc15630b58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp"
Oct 31 13:45:00.429335 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.429311    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67"]
Oct 31 13:45:00.429473 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.429402    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw"
Oct 31 13:45:00.435373 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.435347    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8tkm\" (UniqueName: \"kubernetes.io/projected/680bd885-b3c7-42f5-a3a8-4f2950d0b462-kube-api-access-q8tkm\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp\" (UID: \"680bd885-b3c7-42f5-a3a8-4f2950d0b462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp"
Oct 31 13:45:00.436818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.435485    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6bdt\" (UniqueName: \"kubernetes.io/projected/9daa213c-e1ea-473d-8ed6-a2bc15630b58-kube-api-access-m6bdt\") pod \"maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp\" (UID: \"9daa213c-e1ea-473d-8ed6-a2bc15630b58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp"
Oct 31 13:45:00.436818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.435516    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sfnj\" (UniqueName: \"kubernetes.io/projected/6bb5549e-6008-46a8-8e55-3b37cb62cd60-kube-api-access-8sfnj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8\" (UID: \"6bb5549e-6008-46a8-8e55-3b37cb62cd60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8"
Oct 31 13:45:00.436818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.436590    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw6hh\" (UniqueName: \"kubernetes.io/projected/e615817e-f1b7-485a-b4d4-b63b54b6db48-kube-api-access-qw6hh\") pod \"maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7\" (UID: \"e615817e-f1b7-485a-b4d4-b63b54b6db48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7"
Oct 31 13:45:00.438394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.438371    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw"]
Oct 31 13:45:00.519474 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.519435    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77411480-49b1-49e5-90f0-1fc4416288da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs\" (UID: \"77411480-49b1-49e5-90f0-1fc4416288da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs"
Oct 31 13:45:00.519673 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.519648    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6498d\" (UniqueName: \"kubernetes.io/projected/77411480-49b1-49e5-90f0-1fc4416288da-kube-api-access-6498d\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs\" (UID: \"77411480-49b1-49e5-90f0-1fc4416288da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs"
Oct 31 13:45:00.519734 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.519700    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9485106e-58f9-4b42-ae34-0136f6345af2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4\" (UID: \"9485106e-58f9-4b42-ae34-0136f6345af2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4"
Oct 31 13:45:00.519810 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.519793    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fe9f57-eaa6-49c1-9a63-f37485554a23-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67\" (UID: \"61fe9f57-eaa6-49c1-9a63-f37485554a23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67"
Oct 31 13:45:00.519887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.519830    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcghq\" (UniqueName: \"kubernetes.io/projected/61fe9f57-eaa6-49c1-9a63-f37485554a23-kube-api-access-zcghq\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67\" (UID: \"61fe9f57-eaa6-49c1-9a63-f37485554a23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67"
Oct 31 13:45:00.519950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.519895    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmvrc\" (UniqueName: \"kubernetes.io/projected/9485106e-58f9-4b42-ae34-0136f6345af2-kube-api-access-bmvrc\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4\" (UID: \"9485106e-58f9-4b42-ae34-0136f6345af2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4"
Oct 31 13:45:00.520027 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.519971    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d9f9bc-22ec-4943-8a8c-9ed7abe7af98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8\" (UID: \"55d9f9bc-22ec-4943-8a8c-9ed7abe7af98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8"
Oct 31 13:45:00.520106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.520032    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85ssh\" (UniqueName: \"kubernetes.io/projected/55d9f9bc-22ec-4943-8a8c-9ed7abe7af98-kube-api-access-85ssh\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8\" (UID: \"55d9f9bc-22ec-4943-8a8c-9ed7abe7af98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8"
Oct 31 13:45:00.520106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.520062    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc233cb4-c11a-4bd5-9abb-951884813235-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw\" (UID: \"dc233cb4-c11a-4bd5-9abb-951884813235\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw"
Oct 31 13:45:00.520106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.520099    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xw2h\" (UniqueName: \"kubernetes.io/projected/dc233cb4-c11a-4bd5-9abb-951884813235-kube-api-access-4xw2h\") pod \"maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw\" (UID: \"dc233cb4-c11a-4bd5-9abb-951884813235\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw"
Oct 31 13:45:00.520334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.520306    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77411480-49b1-49e5-90f0-1fc4416288da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs\" (UID: \"77411480-49b1-49e5-90f0-1fc4416288da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs"
Oct 31 13:45:00.520535 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.520514    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fe9f57-eaa6-49c1-9a63-f37485554a23-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67\" (UID: \"61fe9f57-eaa6-49c1-9a63-f37485554a23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67"
Oct 31 13:45:00.520640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.520614    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9485106e-58f9-4b42-ae34-0136f6345af2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4\" (UID: \"9485106e-58f9-4b42-ae34-0136f6345af2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4"
Oct 31 13:45:00.520754 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.520734    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d9f9bc-22ec-4943-8a8c-9ed7abe7af98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8\" (UID: \"55d9f9bc-22ec-4943-8a8c-9ed7abe7af98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8"
Oct 31 13:45:00.524032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.524005    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76"
Oct 31 13:45:00.535905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.535839    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmvrc\" (UniqueName: \"kubernetes.io/projected/9485106e-58f9-4b42-ae34-0136f6345af2-kube-api-access-bmvrc\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4\" (UID: \"9485106e-58f9-4b42-ae34-0136f6345af2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4"
Oct 31 13:45:00.536136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.536106    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85ssh\" (UniqueName: \"kubernetes.io/projected/55d9f9bc-22ec-4943-8a8c-9ed7abe7af98-kube-api-access-85ssh\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8\" (UID: \"55d9f9bc-22ec-4943-8a8c-9ed7abe7af98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8"
Oct 31 13:45:00.536413 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.536383    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcghq\" (UniqueName: \"kubernetes.io/projected/61fe9f57-eaa6-49c1-9a63-f37485554a23-kube-api-access-zcghq\") pod \"maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67\" (UID: \"61fe9f57-eaa6-49c1-9a63-f37485554a23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67"
Oct 31 13:45:00.536778 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.536758    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6498d\" (UniqueName: \"kubernetes.io/projected/77411480-49b1-49e5-90f0-1fc4416288da-kube-api-access-6498d\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs\" (UID: \"77411480-49b1-49e5-90f0-1fc4416288da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs"
Oct 31 13:45:00.556628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.556582    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp"
Oct 31 13:45:00.573961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.573891    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8"
Oct 31 13:45:00.594038 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.594010    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp"
Oct 31 13:45:00.612555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.612525    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7"
Oct 31 13:45:00.623212 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.623070    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc233cb4-c11a-4bd5-9abb-951884813235-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw\" (UID: \"dc233cb4-c11a-4bd5-9abb-951884813235\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw"
Oct 31 13:45:00.623212 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.623129    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xw2h\" (UniqueName: \"kubernetes.io/projected/dc233cb4-c11a-4bd5-9abb-951884813235-kube-api-access-4xw2h\") pod \"maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw\" (UID: \"dc233cb4-c11a-4bd5-9abb-951884813235\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw"
Oct 31 13:45:00.624830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.624801    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc233cb4-c11a-4bd5-9abb-951884813235-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw\" (UID: \"dc233cb4-c11a-4bd5-9abb-951884813235\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw"
Oct 31 13:45:00.632851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.632818    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs"
Oct 31 13:45:00.685167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.683631    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xw2h\" (UniqueName: \"kubernetes.io/projected/dc233cb4-c11a-4bd5-9abb-951884813235-kube-api-access-4xw2h\") pod \"maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw\" (UID: \"dc233cb4-c11a-4bd5-9abb-951884813235\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw"
Oct 31 13:45:00.685167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.683821    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4"
Oct 31 13:45:00.731889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.731197    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8"
Oct 31 13:45:00.733006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.732941    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67"
Oct 31 13:45:00.798971 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:00.790257    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw"
Oct 31 13:45:01.189828 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:01.189794    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f7fe62-925f-48d0-a15e-ed6c4eb56f2b" path="/var/lib/kubelet/pods/30f7fe62-925f-48d0-a15e-ed6c4eb56f2b/volumes"
Oct 31 13:45:01.190940 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:01.190916    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76"]
Oct 31 13:45:01.314229 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:01.313010    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp"]
Oct 31 13:45:01.397778 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:01.397692    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:01.405984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:01.405947    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8"]
Oct 31 13:45:01.588289 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:45:01.588257    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680bd885_b3c7_42f5_a3a8_4f2950d0b462.slice/crio-7c0faf35751b30c038fb4f3c911cceadf8fbd4003e54bb678e78fb486cc03f20 WatchSource:0}: Error finding container 7c0faf35751b30c038fb4f3c911cceadf8fbd4003e54bb678e78fb486cc03f20: Status 404 returned error can't find the container with id 7c0faf35751b30c038fb4f3c911cceadf8fbd4003e54bb678e78fb486cc03f20
Oct 31 13:45:01.589395 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:01.589364    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp"]
Oct 31 13:45:01.612792 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:01.612758    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4"]
Oct 31 13:45:01.614489 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:45:01.614445    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice/crio-f4ff7173cf463fa77294cb5afd3e25bd0fbc5e211a501b20b208a215eb151f31 WatchSource:0}: Error finding container f4ff7173cf463fa77294cb5afd3e25bd0fbc5e211a501b20b208a215eb151f31: Status 404 returned error can't find the container with id f4ff7173cf463fa77294cb5afd3e25bd0fbc5e211a501b20b208a215eb151f31
Oct 31 13:45:01.620328 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:01.620295    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs"]
Oct 31 13:45:01.635747 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:01.635716    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7"]
Oct 31 13:45:01.641214 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:45:01.641190    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode615817e_f1b7_485a_b4d4_b63b54b6db48.slice/crio-56544e22d36ee98b267c5fb95acbc3f8ea7b27725a4b8f49aaf4d27894dfda03 WatchSource:0}: Error finding container 56544e22d36ee98b267c5fb95acbc3f8ea7b27725a4b8f49aaf4d27894dfda03: Status 404 returned error can't find the container with id 56544e22d36ee98b267c5fb95acbc3f8ea7b27725a4b8f49aaf4d27894dfda03
Oct 31 13:45:01.813643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:01.813612    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw"]
Oct 31 13:45:01.818231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:01.818206    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67"]
Oct 31 13:45:01.825320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:01.825298    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8"]
Oct 31 13:45:01.849403 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:45:01.849368    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61fe9f57_eaa6_49c1_9a63_f37485554a23.slice/crio-79fa73e23286bc145712d6d4cde5339cbc17bd2bdcabdd83b13808dd417dc315 WatchSource:0}: Error finding container 79fa73e23286bc145712d6d4cde5339cbc17bd2bdcabdd83b13808dd417dc315: Status 404 returned error can't find the container with id 79fa73e23286bc145712d6d4cde5339cbc17bd2bdcabdd83b13808dd417dc315
Oct 31 13:45:01.850610 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:45:01.850587    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc233cb4_c11a_4bd5_9abb_951884813235.slice/crio-fc6ec93576cca0cf75b916fe4e15e7c3e1320ed9f04ebb244e41aaae753a4684 WatchSource:0}: Error finding container fc6ec93576cca0cf75b916fe4e15e7c3e1320ed9f04ebb244e41aaae753a4684: Status 404 returned error can't find the container with id fc6ec93576cca0cf75b916fe4e15e7c3e1320ed9f04ebb244e41aaae753a4684
Oct 31 13:45:02.191695 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:02.191648    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4" event={"ID":"9485106e-58f9-4b42-ae34-0136f6345af2","Type":"ContainerStarted","Data":"f4ff7173cf463fa77294cb5afd3e25bd0fbc5e211a501b20b208a215eb151f31"}
Oct 31 13:45:02.194021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:02.193985    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8" event={"ID":"55d9f9bc-22ec-4943-8a8c-9ed7abe7af98","Type":"ContainerStarted","Data":"216f3d035980cb6319b9302695212b44327305e649f52dccbe1df92f5133fe03"}
Oct 31 13:45:02.196057 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:02.196027    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8" event={"ID":"6bb5549e-6008-46a8-8e55-3b37cb62cd60","Type":"ContainerStarted","Data":"9ecf1e5a59950ed2fdb00e1fcc26a6b88c97862dc8c91221bb6f9db018502ea8"}
Oct 31 13:45:02.198079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:02.198047    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp" event={"ID":"9daa213c-e1ea-473d-8ed6-a2bc15630b58","Type":"ContainerStarted","Data":"c03166d87cc055a621b8e13fb439de4b3633931a90e6eeddac69a8a694747ad7"}
Oct 31 13:45:02.199957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:02.199929    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw" event={"ID":"dc233cb4-c11a-4bd5-9abb-951884813235","Type":"ContainerStarted","Data":"fc6ec93576cca0cf75b916fe4e15e7c3e1320ed9f04ebb244e41aaae753a4684"}
Oct 31 13:45:02.201706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:02.201681    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs" event={"ID":"77411480-49b1-49e5-90f0-1fc4416288da","Type":"ContainerStarted","Data":"4a68214c6cfeb32c88fe5e6c59ee3be86094019df5ca2fc8f4fd6f81b79c0dae"}
Oct 31 13:45:02.204319 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:02.204285    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76" event={"ID":"d507266c-1d74-4d2c-9af8-c9990fe98174","Type":"ContainerStarted","Data":"f1339cefd7059e4d3ec293ddcab6a9b5a2e00796ffd6d689389710b6ca70c0a7"}
Oct 31 13:45:02.204438 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:02.204347    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76" event={"ID":"d507266c-1d74-4d2c-9af8-c9990fe98174","Type":"ContainerStarted","Data":"683b298b886da9b556fce5954f65a1873047e87a5e1ba560856c29b0831a47f0"}
Oct 31 13:45:02.206138 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:02.206073    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp" event={"ID":"680bd885-b3c7-42f5-a3a8-4f2950d0b462","Type":"ContainerStarted","Data":"7c0faf35751b30c038fb4f3c911cceadf8fbd4003e54bb678e78fb486cc03f20"}
Oct 31 13:45:02.207614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:02.207591    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7" event={"ID":"e615817e-f1b7-485a-b4d4-b63b54b6db48","Type":"ContainerStarted","Data":"56544e22d36ee98b267c5fb95acbc3f8ea7b27725a4b8f49aaf4d27894dfda03"}
Oct 31 13:45:02.209086 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:02.209061    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67" event={"ID":"61fe9f57-eaa6-49c1-9a63-f37485554a23","Type":"ContainerStarted","Data":"79fa73e23286bc145712d6d4cde5339cbc17bd2bdcabdd83b13808dd417dc315"}
Oct 31 13:45:02.233393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:02.233340    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76" podStartSLOduration=2.233324788 podStartE2EDuration="2.233324788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:02.229939872 +0000 UTC m=+2499.628288498" watchObservedRunningTime="2026-10-31 13:45:02.233324788 +0000 UTC m=+2499.631673451"
Oct 31 13:45:02.513005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:02.512966    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:03.224384 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.224346    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7" event={"ID":"e615817e-f1b7-485a-b4d4-b63b54b6db48","Type":"ContainerStarted","Data":"633c7eb3996ccc54ac0802d47fd24771f4aea0f345d0e93526b86ee3fb76a91b"}
Oct 31 13:45:03.226370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.226343    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67" event={"ID":"61fe9f57-eaa6-49c1-9a63-f37485554a23","Type":"ContainerStarted","Data":"c7ca80332a7fce844c8531ceced055f326bda2fa76e3f3ffd71766d3d62f770b"}
Oct 31 13:45:03.228229 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.228201    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4" event={"ID":"9485106e-58f9-4b42-ae34-0136f6345af2","Type":"ContainerStarted","Data":"64d3b6dda3790e5ec3f4fd9a49f9d55a85bb0e6287aa5330642c87856f4ff3b9"}
Oct 31 13:45:03.229909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.229853    4428 generic.go:356] "Generic (PLEG): container finished" podID="55d9f9bc-22ec-4943-8a8c-9ed7abe7af98" containerID="867c87350441350c88b5893f3982109639514a8a669b6531e438eb7b8c7b7ebe" exitCode=60
Oct 31 13:45:03.230015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.229900    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8" event={"ID":"55d9f9bc-22ec-4943-8a8c-9ed7abe7af98","Type":"ContainerDied","Data":"867c87350441350c88b5893f3982109639514a8a669b6531e438eb7b8c7b7ebe"}
Oct 31 13:45:03.230154 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.230139    4428 scope.go:122] "RemoveContainer" containerID="867c87350441350c88b5893f3982109639514a8a669b6531e438eb7b8c7b7ebe"
Oct 31 13:45:03.232551 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.232529    4428 generic.go:356] "Generic (PLEG): container finished" podID="6bb5549e-6008-46a8-8e55-3b37cb62cd60" containerID="743eed0db405440e3502205b6ecbc316c2c1fa0a01c200fcf585a3c86fcde9ac" exitCode=60
Oct 31 13:45:03.232665 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.232596    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8" event={"ID":"6bb5549e-6008-46a8-8e55-3b37cb62cd60","Type":"ContainerDied","Data":"743eed0db405440e3502205b6ecbc316c2c1fa0a01c200fcf585a3c86fcde9ac"}
Oct 31 13:45:03.232898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.232860    4428 scope.go:122] "RemoveContainer" containerID="743eed0db405440e3502205b6ecbc316c2c1fa0a01c200fcf585a3c86fcde9ac"
Oct 31 13:45:03.234470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.234448    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp" event={"ID":"9daa213c-e1ea-473d-8ed6-a2bc15630b58","Type":"ContainerStarted","Data":"0ad59b8db03d465f41329ea30f2ba7f21e7da6aa7afbb90773a89d30708391c0"}
Oct 31 13:45:03.236253 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.236230    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw" event={"ID":"dc233cb4-c11a-4bd5-9abb-951884813235","Type":"ContainerStarted","Data":"76cdc0575a70a4f423e9a2f34d553c017b8614fbd28140eb883844f34f814c72"}
Oct 31 13:45:03.237932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.237911    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs" event={"ID":"77411480-49b1-49e5-90f0-1fc4416288da","Type":"ContainerStarted","Data":"3155cb914738a7c55561eb7c5f2361ebdc0033ddee85fcdb3f7b7f5133625e7d"}
Oct 31 13:45:03.239584 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.239563    4428 generic.go:356] "Generic (PLEG): container finished" podID="d507266c-1d74-4d2c-9af8-c9990fe98174" containerID="f1339cefd7059e4d3ec293ddcab6a9b5a2e00796ffd6d689389710b6ca70c0a7" exitCode=60
Oct 31 13:45:03.239691 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.239647    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76" event={"ID":"d507266c-1d74-4d2c-9af8-c9990fe98174","Type":"ContainerDied","Data":"f1339cefd7059e4d3ec293ddcab6a9b5a2e00796ffd6d689389710b6ca70c0a7"}
Oct 31 13:45:03.239921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.239839    4428 scope.go:122] "RemoveContainer" containerID="f1339cefd7059e4d3ec293ddcab6a9b5a2e00796ffd6d689389710b6ca70c0a7"
Oct 31 13:45:03.241391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.241364    4428 generic.go:356] "Generic (PLEG): container finished" podID="680bd885-b3c7-42f5-a3a8-4f2950d0b462" containerID="b897aa05a92fea0943b58d11f73a56086d9f4b17a945ab35af2a91ad44f9c70b" exitCode=60
Oct 31 13:45:03.241501 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.241451    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp" event={"ID":"680bd885-b3c7-42f5-a3a8-4f2950d0b462","Type":"ContainerDied","Data":"b897aa05a92fea0943b58d11f73a56086d9f4b17a945ab35af2a91ad44f9c70b"}
Oct 31 13:45:03.241698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.241680    4428 scope.go:122] "RemoveContainer" containerID="b897aa05a92fea0943b58d11f73a56086d9f4b17a945ab35af2a91ad44f9c70b"
Oct 31 13:45:03.258223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.258175    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7" podStartSLOduration=3.258158963 podStartE2EDuration="3.258158963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:03.254432335 +0000 UTC m=+2500.652780956" watchObservedRunningTime="2026-10-31 13:45:03.258158963 +0000 UTC m=+2500.656507587"
Oct 31 13:45:03.279945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.279894    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp" podStartSLOduration=3.2798625 podStartE2EDuration="3.2798625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:03.275004566 +0000 UTC m=+2500.673353192" watchObservedRunningTime="2026-10-31 13:45:03.2798625 +0000 UTC m=+2500.678211123"
Oct 31 13:45:03.471730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.471673    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67" podStartSLOduration=3.471658322 podStartE2EDuration="3.471658322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:03.465353424 +0000 UTC m=+2500.863702048" watchObservedRunningTime="2026-10-31 13:45:03.471658322 +0000 UTC m=+2500.870006982"
Oct 31 13:45:03.486724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.486669    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4" podStartSLOduration=3.486651125 podStartE2EDuration="3.486651125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:03.485516584 +0000 UTC m=+2500.883865212" watchObservedRunningTime="2026-10-31 13:45:03.486651125 +0000 UTC m=+2500.884999750"
Oct 31 13:45:03.514832 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.514780    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw" podStartSLOduration=3.514764519 podStartE2EDuration="3.514764519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:03.510825441 +0000 UTC m=+2500.909174082" watchObservedRunningTime="2026-10-31 13:45:03.514764519 +0000 UTC m=+2500.913113204"
Oct 31 13:45:03.589051 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.584909    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs" podStartSLOduration=3.5848882570000002 podStartE2EDuration="3.584888257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:03.583707542 +0000 UTC m=+2500.982056167" watchObservedRunningTime="2026-10-31 13:45:03.584888257 +0000 UTC m=+2500.983236882"
Oct 31 13:45:03.616183 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:03.609267    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:04.260143 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.260110    4428 generic.go:356] "Generic (PLEG): container finished" podID="e615817e-f1b7-485a-b4d4-b63b54b6db48" containerID="633c7eb3996ccc54ac0802d47fd24771f4aea0f345d0e93526b86ee3fb76a91b" exitCode=60
Oct 31 13:45:04.260288 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.260194    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7" event={"ID":"e615817e-f1b7-485a-b4d4-b63b54b6db48","Type":"ContainerDied","Data":"633c7eb3996ccc54ac0802d47fd24771f4aea0f345d0e93526b86ee3fb76a91b"}
Oct 31 13:45:04.260668 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.260644    4428 scope.go:122] "RemoveContainer" containerID="633c7eb3996ccc54ac0802d47fd24771f4aea0f345d0e93526b86ee3fb76a91b"
Oct 31 13:45:04.262943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.262528    4428 generic.go:356] "Generic (PLEG): container finished" podID="61fe9f57-eaa6-49c1-9a63-f37485554a23" containerID="c7ca80332a7fce844c8531ceced055f326bda2fa76e3f3ffd71766d3d62f770b" exitCode=60
Oct 31 13:45:04.262943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.262607    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67" event={"ID":"61fe9f57-eaa6-49c1-9a63-f37485554a23","Type":"ContainerDied","Data":"c7ca80332a7fce844c8531ceced055f326bda2fa76e3f3ffd71766d3d62f770b"}
Oct 31 13:45:04.264156 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.264131    4428 scope.go:122] "RemoveContainer" containerID="c7ca80332a7fce844c8531ceced055f326bda2fa76e3f3ffd71766d3d62f770b"
Oct 31 13:45:04.266116 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.265989    4428 generic.go:356] "Generic (PLEG): container finished" podID="9485106e-58f9-4b42-ae34-0136f6345af2" containerID="64d3b6dda3790e5ec3f4fd9a49f9d55a85bb0e6287aa5330642c87856f4ff3b9" exitCode=60
Oct 31 13:45:04.266116 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.266065    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4" event={"ID":"9485106e-58f9-4b42-ae34-0136f6345af2","Type":"ContainerDied","Data":"64d3b6dda3790e5ec3f4fd9a49f9d55a85bb0e6287aa5330642c87856f4ff3b9"}
Oct 31 13:45:04.266375 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.266356    4428 scope.go:122] "RemoveContainer" containerID="64d3b6dda3790e5ec3f4fd9a49f9d55a85bb0e6287aa5330642c87856f4ff3b9"
Oct 31 13:45:04.269657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.269538    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8" event={"ID":"55d9f9bc-22ec-4943-8a8c-9ed7abe7af98","Type":"ContainerStarted","Data":"d3908faeff05e0b906b31adfea188260f2250418efad81f9d9a7e2b0e21dba40"}
Oct 31 13:45:04.271646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.271627    4428 generic.go:356] "Generic (PLEG): container finished" podID="9daa213c-e1ea-473d-8ed6-a2bc15630b58" containerID="0ad59b8db03d465f41329ea30f2ba7f21e7da6aa7afbb90773a89d30708391c0" exitCode=60
Oct 31 13:45:04.271724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.271694    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp" event={"ID":"9daa213c-e1ea-473d-8ed6-a2bc15630b58","Type":"ContainerDied","Data":"0ad59b8db03d465f41329ea30f2ba7f21e7da6aa7afbb90773a89d30708391c0"}
Oct 31 13:45:04.272023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.271997    4428 scope.go:122] "RemoveContainer" containerID="0ad59b8db03d465f41329ea30f2ba7f21e7da6aa7afbb90773a89d30708391c0"
Oct 31 13:45:04.273734 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.273711    4428 generic.go:356] "Generic (PLEG): container finished" podID="dc233cb4-c11a-4bd5-9abb-951884813235" containerID="76cdc0575a70a4f423e9a2f34d553c017b8614fbd28140eb883844f34f814c72" exitCode=60
Oct 31 13:45:04.273839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.273739    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw" event={"ID":"dc233cb4-c11a-4bd5-9abb-951884813235","Type":"ContainerDied","Data":"76cdc0575a70a4f423e9a2f34d553c017b8614fbd28140eb883844f34f814c72"}
Oct 31 13:45:04.274005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.273991    4428 scope.go:122] "RemoveContainer" containerID="76cdc0575a70a4f423e9a2f34d553c017b8614fbd28140eb883844f34f814c72"
Oct 31 13:45:04.275663 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.275642    4428 generic.go:356] "Generic (PLEG): container finished" podID="77411480-49b1-49e5-90f0-1fc4416288da" containerID="3155cb914738a7c55561eb7c5f2361ebdc0033ddee85fcdb3f7b7f5133625e7d" exitCode=60
Oct 31 13:45:04.275756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.275722    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs" event={"ID":"77411480-49b1-49e5-90f0-1fc4416288da","Type":"ContainerDied","Data":"3155cb914738a7c55561eb7c5f2361ebdc0033ddee85fcdb3f7b7f5133625e7d"}
Oct 31 13:45:04.275940 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.275925    4428 scope.go:122] "RemoveContainer" containerID="3155cb914738a7c55561eb7c5f2361ebdc0033ddee85fcdb3f7b7f5133625e7d"
Oct 31 13:45:04.407935 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.407890    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8" podStartSLOduration=4.407859847 podStartE2EDuration="4.407859847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:04.404700966 +0000 UTC m=+2501.803049590" watchObservedRunningTime="2026-10-31 13:45:04.407859847 +0000 UTC m=+2501.806208471"
Oct 31 13:45:04.737285 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:04.737240    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:05.294624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.294581    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw" event={"ID":"dc233cb4-c11a-4bd5-9abb-951884813235","Type":"ContainerStarted","Data":"731aa92bbb49c5ebd93e4b7a8ca5b8ce99fd8e48fc09b76f481dd0feb5a957e3"}
Oct 31 13:45:05.298224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.298184    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs" event={"ID":"77411480-49b1-49e5-90f0-1fc4416288da","Type":"ContainerStarted","Data":"ff473a4d21cc30d7cd596ca6c96bd79c6d39b0ecaf4690c570084fbb3516235f"}
Oct 31 13:45:05.302406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.302374    4428 generic.go:356] "Generic (PLEG): container finished" podID="d507266c-1d74-4d2c-9af8-c9990fe98174" containerID="40ee909b2cb40195dcfcb96dbcf4f9284db502e2522c2e6a01135c9706904ddf" exitCode=60
Oct 31 13:45:05.302557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.302455    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76" event={"ID":"d507266c-1d74-4d2c-9af8-c9990fe98174","Type":"ContainerDied","Data":"40ee909b2cb40195dcfcb96dbcf4f9284db502e2522c2e6a01135c9706904ddf"}
Oct 31 13:45:05.302557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.302510    4428 scope.go:122] "RemoveContainer" containerID="f1339cefd7059e4d3ec293ddcab6a9b5a2e00796ffd6d689389710b6ca70c0a7"
Oct 31 13:45:05.302804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.302785    4428 scope.go:122] "RemoveContainer" containerID="40ee909b2cb40195dcfcb96dbcf4f9284db502e2522c2e6a01135c9706904ddf"
Oct 31 13:45:05.303167 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:05.303139    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76_odh-ai-gateway-infra(d507266c-1d74-4d2c-9af8-c9990fe98174)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76" podUID="d507266c-1d74-4d2c-9af8-c9990fe98174"
Oct 31 13:45:05.305027 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.305006    4428 generic.go:356] "Generic (PLEG): container finished" podID="680bd885-b3c7-42f5-a3a8-4f2950d0b462" containerID="9fa384df2d11aebb2fba2360adc408bf7f9aa7767046215aceb69aa61e821542" exitCode=60
Oct 31 13:45:05.305118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.305069    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp" event={"ID":"680bd885-b3c7-42f5-a3a8-4f2950d0b462","Type":"ContainerDied","Data":"9fa384df2d11aebb2fba2360adc408bf7f9aa7767046215aceb69aa61e821542"}
Oct 31 13:45:05.305389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.305372    4428 scope.go:122] "RemoveContainer" containerID="9fa384df2d11aebb2fba2360adc408bf7f9aa7767046215aceb69aa61e821542"
Oct 31 13:45:05.305720 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:05.305699    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp_odh-ai-gateway-infra(680bd885-b3c7-42f5-a3a8-4f2950d0b462)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp" podUID="680bd885-b3c7-42f5-a3a8-4f2950d0b462"
Oct 31 13:45:05.307318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.307289    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7" event={"ID":"e615817e-f1b7-485a-b4d4-b63b54b6db48","Type":"ContainerStarted","Data":"72afb27e590373ac7f90f3bc05927562a4b4661a9b07a93920bd9997fe37c26d"}
Oct 31 13:45:05.309404 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.309376    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67" event={"ID":"61fe9f57-eaa6-49c1-9a63-f37485554a23","Type":"ContainerStarted","Data":"f191504f3d381562535e6a2cb2e90ba2f9769a15907ec81e4c6d47ca120f99d3"}
Oct 31 13:45:05.312684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.312660    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4" event={"ID":"9485106e-58f9-4b42-ae34-0136f6345af2","Type":"ContainerStarted","Data":"1edc619b8b375f711577f4b1b0ac0e588cc1908aa39a2e30ddf64279c99b2091"}
Oct 31 13:45:05.314983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.314950    4428 generic.go:356] "Generic (PLEG): container finished" podID="55d9f9bc-22ec-4943-8a8c-9ed7abe7af98" containerID="d3908faeff05e0b906b31adfea188260f2250418efad81f9d9a7e2b0e21dba40" exitCode=60
Oct 31 13:45:05.315115 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.315043    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8" event={"ID":"55d9f9bc-22ec-4943-8a8c-9ed7abe7af98","Type":"ContainerDied","Data":"d3908faeff05e0b906b31adfea188260f2250418efad81f9d9a7e2b0e21dba40"}
Oct 31 13:45:05.315502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.315459    4428 scope.go:122] "RemoveContainer" containerID="d3908faeff05e0b906b31adfea188260f2250418efad81f9d9a7e2b0e21dba40"
Oct 31 13:45:05.318167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.318143    4428 generic.go:356] "Generic (PLEG): container finished" podID="6bb5549e-6008-46a8-8e55-3b37cb62cd60" containerID="07229caea95318bbdfc954a269a093763ffcfce6dbffa44422cbe87775b61d8a" exitCode=60
Oct 31 13:45:05.318437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.318403    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8" event={"ID":"6bb5549e-6008-46a8-8e55-3b37cb62cd60","Type":"ContainerDied","Data":"07229caea95318bbdfc954a269a093763ffcfce6dbffa44422cbe87775b61d8a"}
Oct 31 13:45:05.318601 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.318581    4428 scope.go:122] "RemoveContainer" containerID="07229caea95318bbdfc954a269a093763ffcfce6dbffa44422cbe87775b61d8a"
Oct 31 13:45:05.319004 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:05.318981    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8_odh-ai-gateway-infra(6bb5549e-6008-46a8-8e55-3b37cb62cd60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8" podUID="6bb5549e-6008-46a8-8e55-3b37cb62cd60"
Oct 31 13:45:05.320667 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.320644    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp" event={"ID":"9daa213c-e1ea-473d-8ed6-a2bc15630b58","Type":"ContainerStarted","Data":"defb4dc60bf40c064b39850a66f6f0eec5aff01a482580c0e2c81137a4b1d116"}
Oct 31 13:45:05.320994 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:05.320966    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8_odh-ai-gateway-infra(55d9f9bc-22ec-4943-8a8c-9ed7abe7af98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8" podUID="55d9f9bc-22ec-4943-8a8c-9ed7abe7af98"
Oct 31 13:45:05.321679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.321656    4428 scope.go:122] "RemoveContainer" containerID="b897aa05a92fea0943b58d11f73a56086d9f4b17a945ab35af2a91ad44f9c70b"
Oct 31 13:45:05.337362 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.337340    4428 scope.go:122] "RemoveContainer" containerID="867c87350441350c88b5893f3982109639514a8a669b6531e438eb7b8c7b7ebe"
Oct 31 13:45:05.359071 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.359043    4428 scope.go:122] "RemoveContainer" containerID="743eed0db405440e3502205b6ecbc316c2c1fa0a01c200fcf585a3c86fcde9ac"
Oct 31 13:45:05.732226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:05.732078    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:06.056900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.055625    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:06.160379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.160340    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s" podUID="dd12e2cb-e569-446e-8cbf-d6cd4779c833" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:45:06.160546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.160342    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p" podUID="9cf05b3f-e7f7-4b17-b06c-c015ae338bad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:45:06.342945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.342807    4428 generic.go:356] "Generic (PLEG): container finished" podID="61fe9f57-eaa6-49c1-9a63-f37485554a23" containerID="f191504f3d381562535e6a2cb2e90ba2f9769a15907ec81e4c6d47ca120f99d3" exitCode=60
Oct 31 13:45:06.342945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.342903    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67" event={"ID":"61fe9f57-eaa6-49c1-9a63-f37485554a23","Type":"ContainerDied","Data":"f191504f3d381562535e6a2cb2e90ba2f9769a15907ec81e4c6d47ca120f99d3"}
Oct 31 13:45:06.343177 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.342982    4428 scope.go:122] "RemoveContainer" containerID="c7ca80332a7fce844c8531ceced055f326bda2fa76e3f3ffd71766d3d62f770b"
Oct 31 13:45:06.343341 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.343322    4428 scope.go:122] "RemoveContainer" containerID="f191504f3d381562535e6a2cb2e90ba2f9769a15907ec81e4c6d47ca120f99d3"
Oct 31 13:45:06.343808 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:06.343778    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67_odh-ai-gateway-infra(61fe9f57-eaa6-49c1-9a63-f37485554a23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67" podUID="61fe9f57-eaa6-49c1-9a63-f37485554a23"
Oct 31 13:45:06.345752 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.345725    4428 generic.go:356] "Generic (PLEG): container finished" podID="9485106e-58f9-4b42-ae34-0136f6345af2" containerID="1edc619b8b375f711577f4b1b0ac0e588cc1908aa39a2e30ddf64279c99b2091" exitCode=60
Oct 31 13:45:06.345930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.345801    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4" event={"ID":"9485106e-58f9-4b42-ae34-0136f6345af2","Type":"ContainerDied","Data":"1edc619b8b375f711577f4b1b0ac0e588cc1908aa39a2e30ddf64279c99b2091"}
Oct 31 13:45:06.346231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.346213    4428 scope.go:122] "RemoveContainer" containerID="1edc619b8b375f711577f4b1b0ac0e588cc1908aa39a2e30ddf64279c99b2091"
Oct 31 13:45:06.346649 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:06.346626    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4_odh-ai-gateway-infra(9485106e-58f9-4b42-ae34-0136f6345af2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4" podUID="9485106e-58f9-4b42-ae34-0136f6345af2"
Oct 31 13:45:06.348252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.348229    4428 scope.go:122] "RemoveContainer" containerID="d3908faeff05e0b906b31adfea188260f2250418efad81f9d9a7e2b0e21dba40"
Oct 31 13:45:06.348657 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:06.348633    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8_odh-ai-gateway-infra(55d9f9bc-22ec-4943-8a8c-9ed7abe7af98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8" podUID="55d9f9bc-22ec-4943-8a8c-9ed7abe7af98"
Oct 31 13:45:06.351803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.350829    4428 scope.go:122] "RemoveContainer" containerID="07229caea95318bbdfc954a269a093763ffcfce6dbffa44422cbe87775b61d8a"
Oct 31 13:45:06.351803 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:06.351277    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8_odh-ai-gateway-infra(6bb5549e-6008-46a8-8e55-3b37cb62cd60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8" podUID="6bb5549e-6008-46a8-8e55-3b37cb62cd60"
Oct 31 13:45:06.355032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.355006    4428 generic.go:356] "Generic (PLEG): container finished" podID="9daa213c-e1ea-473d-8ed6-a2bc15630b58" containerID="defb4dc60bf40c064b39850a66f6f0eec5aff01a482580c0e2c81137a4b1d116" exitCode=60
Oct 31 13:45:06.355172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.355153    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp" event={"ID":"9daa213c-e1ea-473d-8ed6-a2bc15630b58","Type":"ContainerDied","Data":"defb4dc60bf40c064b39850a66f6f0eec5aff01a482580c0e2c81137a4b1d116"}
Oct 31 13:45:06.355614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.355595    4428 scope.go:122] "RemoveContainer" containerID="defb4dc60bf40c064b39850a66f6f0eec5aff01a482580c0e2c81137a4b1d116"
Oct 31 13:45:06.356055 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:06.356027    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp_odh-ai-gateway-infra(9daa213c-e1ea-473d-8ed6-a2bc15630b58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp" podUID="9daa213c-e1ea-473d-8ed6-a2bc15630b58"
Oct 31 13:45:06.358689 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.358660    4428 generic.go:356] "Generic (PLEG): container finished" podID="dc233cb4-c11a-4bd5-9abb-951884813235" containerID="731aa92bbb49c5ebd93e4b7a8ca5b8ce99fd8e48fc09b76f481dd0feb5a957e3" exitCode=60
Oct 31 13:45:06.358807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.358749    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw" event={"ID":"dc233cb4-c11a-4bd5-9abb-951884813235","Type":"ContainerDied","Data":"731aa92bbb49c5ebd93e4b7a8ca5b8ce99fd8e48fc09b76f481dd0feb5a957e3"}
Oct 31 13:45:06.359072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.359056    4428 scope.go:122] "RemoveContainer" containerID="731aa92bbb49c5ebd93e4b7a8ca5b8ce99fd8e48fc09b76f481dd0feb5a957e3"
Oct 31 13:45:06.359491 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:06.359450    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw_odh-ai-gateway-infra(dc233cb4-c11a-4bd5-9abb-951884813235)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw" podUID="dc233cb4-c11a-4bd5-9abb-951884813235"
Oct 31 13:45:06.361671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.361647    4428 generic.go:356] "Generic (PLEG): container finished" podID="77411480-49b1-49e5-90f0-1fc4416288da" containerID="ff473a4d21cc30d7cd596ca6c96bd79c6d39b0ecaf4690c570084fbb3516235f" exitCode=60
Oct 31 13:45:06.361819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.361798    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs" event={"ID":"77411480-49b1-49e5-90f0-1fc4416288da","Type":"ContainerDied","Data":"ff473a4d21cc30d7cd596ca6c96bd79c6d39b0ecaf4690c570084fbb3516235f"}
Oct 31 13:45:06.362611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.362148    4428 scope.go:122] "RemoveContainer" containerID="ff473a4d21cc30d7cd596ca6c96bd79c6d39b0ecaf4690c570084fbb3516235f"
Oct 31 13:45:06.363029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.362996    4428 scope.go:122] "RemoveContainer" containerID="64d3b6dda3790e5ec3f4fd9a49f9d55a85bb0e6287aa5330642c87856f4ff3b9"
Oct 31 13:45:06.363832 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:06.363809    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs_odh-ai-gateway-infra(77411480-49b1-49e5-90f0-1fc4416288da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs" podUID="77411480-49b1-49e5-90f0-1fc4416288da"
Oct 31 13:45:06.365012 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.364995    4428 scope.go:122] "RemoveContainer" containerID="40ee909b2cb40195dcfcb96dbcf4f9284db502e2522c2e6a01135c9706904ddf"
Oct 31 13:45:06.365340 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:06.365318    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76_odh-ai-gateway-infra(d507266c-1d74-4d2c-9af8-c9990fe98174)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76" podUID="d507266c-1d74-4d2c-9af8-c9990fe98174"
Oct 31 13:45:06.369042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.368824    4428 scope.go:122] "RemoveContainer" containerID="9fa384df2d11aebb2fba2360adc408bf7f9aa7767046215aceb69aa61e821542"
Oct 31 13:45:06.370090 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:06.370043    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp_odh-ai-gateway-infra(680bd885-b3c7-42f5-a3a8-4f2950d0b462)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp" podUID="680bd885-b3c7-42f5-a3a8-4f2950d0b462"
Oct 31 13:45:06.372960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.372936    4428 generic.go:356] "Generic (PLEG): container finished" podID="e615817e-f1b7-485a-b4d4-b63b54b6db48" containerID="72afb27e590373ac7f90f3bc05927562a4b4661a9b07a93920bd9997fe37c26d" exitCode=60
Oct 31 13:45:06.373110 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.372985    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7" event={"ID":"e615817e-f1b7-485a-b4d4-b63b54b6db48","Type":"ContainerDied","Data":"72afb27e590373ac7f90f3bc05927562a4b4661a9b07a93920bd9997fe37c26d"}
Oct 31 13:45:06.373305 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.373287    4428 scope.go:122] "RemoveContainer" containerID="72afb27e590373ac7f90f3bc05927562a4b4661a9b07a93920bd9997fe37c26d"
Oct 31 13:45:06.373645 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:06.373623    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7_odh-ai-gateway-infra(e615817e-f1b7-485a-b4d4-b63b54b6db48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7" podUID="e615817e-f1b7-485a-b4d4-b63b54b6db48"
Oct 31 13:45:06.386776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.386750    4428 scope.go:122] "RemoveContainer" containerID="0ad59b8db03d465f41329ea30f2ba7f21e7da6aa7afbb90773a89d30708391c0"
Oct 31 13:45:06.412913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.412851    4428 scope.go:122] "RemoveContainer" containerID="76cdc0575a70a4f423e9a2f34d553c017b8614fbd28140eb883844f34f814c72"
Oct 31 13:45:06.437106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.437080    4428 scope.go:122] "RemoveContainer" containerID="3155cb914738a7c55561eb7c5f2361ebdc0033ddee85fcdb3f7b7f5133625e7d"
Oct 31 13:45:06.463251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.463185    4428 scope.go:122] "RemoveContainer" containerID="633c7eb3996ccc54ac0802d47fd24771f4aea0f345d0e93526b86ee3fb76a91b"
Oct 31 13:45:06.958897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:06.955941    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:07.384002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:07.383972    4428 scope.go:122] "RemoveContainer" containerID="defb4dc60bf40c064b39850a66f6f0eec5aff01a482580c0e2c81137a4b1d116"
Oct 31 13:45:07.384502 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:07.384448    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp_odh-ai-gateway-infra(9daa213c-e1ea-473d-8ed6-a2bc15630b58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp" podUID="9daa213c-e1ea-473d-8ed6-a2bc15630b58"
Oct 31 13:45:07.385824 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:07.385803    4428 scope.go:122] "RemoveContainer" containerID="731aa92bbb49c5ebd93e4b7a8ca5b8ce99fd8e48fc09b76f481dd0feb5a957e3"
Oct 31 13:45:07.386281 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:07.386252    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw_odh-ai-gateway-infra(dc233cb4-c11a-4bd5-9abb-951884813235)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw" podUID="dc233cb4-c11a-4bd5-9abb-951884813235"
Oct 31 13:45:07.387949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:07.387927    4428 scope.go:122] "RemoveContainer" containerID="ff473a4d21cc30d7cd596ca6c96bd79c6d39b0ecaf4690c570084fbb3516235f"
Oct 31 13:45:07.388274 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:07.388250    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs_odh-ai-gateway-infra(77411480-49b1-49e5-90f0-1fc4416288da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs" podUID="77411480-49b1-49e5-90f0-1fc4416288da"
Oct 31 13:45:07.390106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:07.390084    4428 scope.go:122] "RemoveContainer" containerID="72afb27e590373ac7f90f3bc05927562a4b4661a9b07a93920bd9997fe37c26d"
Oct 31 13:45:07.390408 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:07.390389    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7_odh-ai-gateway-infra(e615817e-f1b7-485a-b4d4-b63b54b6db48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7" podUID="e615817e-f1b7-485a-b4d4-b63b54b6db48"
Oct 31 13:45:07.391851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:07.391832    4428 scope.go:122] "RemoveContainer" containerID="f191504f3d381562535e6a2cb2e90ba2f9769a15907ec81e4c6d47ca120f99d3"
Oct 31 13:45:07.392196 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:07.392172    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67_odh-ai-gateway-infra(61fe9f57-eaa6-49c1-9a63-f37485554a23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67" podUID="61fe9f57-eaa6-49c1-9a63-f37485554a23"
Oct 31 13:45:07.393675 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:07.393659    4428 scope.go:122] "RemoveContainer" containerID="1edc619b8b375f711577f4b1b0ac0e588cc1908aa39a2e30ddf64279c99b2091"
Oct 31 13:45:07.393969 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:07.393949    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4_odh-ai-gateway-infra(9485106e-58f9-4b42-ae34-0136f6345af2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4" podUID="9485106e-58f9-4b42-ae34-0136f6345af2"
Oct 31 13:45:07.842786 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:07.840488    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:08.328901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.328875    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"
Oct 31 13:45:08.378990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.378938    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf05b3f-e7f7-4b17-b06c-c015ae338bad-kube-api-access-ngcnv\" (UniqueName: \"kubernetes.io/projected/9cf05b3f-e7f7-4b17-b06c-c015ae338bad-kube-api-access-ngcnv\") pod \"9cf05b3f-e7f7-4b17-b06c-c015ae338bad\" (UID: \"9cf05b3f-e7f7-4b17-b06c-c015ae338bad\") "
Oct 31 13:45:08.379685 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.379658    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cf05b3f-e7f7-4b17-b06c-c015ae338bad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf05b3f-e7f7-4b17-b06c-c015ae338bad-plugins-config-volume\") pod \"9cf05b3f-e7f7-4b17-b06c-c015ae338bad\" (UID: \"9cf05b3f-e7f7-4b17-b06c-c015ae338bad\") "
Oct 31 13:45:08.380178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.380149    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cf05b3f-e7f7-4b17-b06c-c015ae338bad-plugins-config-volume" pod "9cf05b3f-e7f7-4b17-b06c-c015ae338bad" (UID: "9cf05b3f-e7f7-4b17-b06c-c015ae338bad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:08.380724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.380699    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf05b3f-e7f7-4b17-b06c-c015ae338bad-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:08.382354 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.382315    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf05b3f-e7f7-4b17-b06c-c015ae338bad-kube-api-access-ngcnv" pod "9cf05b3f-e7f7-4b17-b06c-c015ae338bad" (UID: "9cf05b3f-e7f7-4b17-b06c-c015ae338bad"). InnerVolumeSpecName "kube-api-access-ngcnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:08.395172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.395141    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"
Oct 31 13:45:08.404238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.404211    4428 generic.go:356] "Generic (PLEG): container finished" podID="9cf05b3f-e7f7-4b17-b06c-c015ae338bad" containerID="f4af763bfc4b28c4c6b9ad53de8620dee77c08acb22ba5930134797bed7dc5e6" exitCode=0
Oct 31 13:45:08.404397 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.404262    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"
Oct 31 13:45:08.404397 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.404293    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p" event={"ID":"9cf05b3f-e7f7-4b17-b06c-c015ae338bad","Type":"ContainerDied","Data":"f4af763bfc4b28c4c6b9ad53de8620dee77c08acb22ba5930134797bed7dc5e6"}
Oct 31 13:45:08.404397 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.404345    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p" event={"ID":"9cf05b3f-e7f7-4b17-b06c-c015ae338bad","Type":"ContainerDied","Data":"a7192d195c0b589768977ad1c1d54efb8fb900b34f46ca47153420b5c5c22c6e"}
Oct 31 13:45:08.404397 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.404373    4428 scope.go:122] "RemoveContainer" containerID="f4af763bfc4b28c4c6b9ad53de8620dee77c08acb22ba5930134797bed7dc5e6"
Oct 31 13:45:08.406037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.406009    4428 generic.go:356] "Generic (PLEG): container finished" podID="dd12e2cb-e569-446e-8cbf-d6cd4779c833" containerID="765142255aff4fff167f6035389dd098a29fed8cb1425591e6107036ff29c1ee" exitCode=0
Oct 31 13:45:08.406160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.406070    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"
Oct 31 13:45:08.406160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.406097    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s" event={"ID":"dd12e2cb-e569-446e-8cbf-d6cd4779c833","Type":"ContainerDied","Data":"765142255aff4fff167f6035389dd098a29fed8cb1425591e6107036ff29c1ee"}
Oct 31 13:45:08.406160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.406143    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s" event={"ID":"dd12e2cb-e569-446e-8cbf-d6cd4779c833","Type":"ContainerDied","Data":"9d118625d7c48e18413fff1a918b002218ca122219ad33ddbeff6f8ac7601e79"}
Oct 31 13:45:08.421758 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.421732    4428 scope.go:122] "RemoveContainer" containerID="f4af763bfc4b28c4c6b9ad53de8620dee77c08acb22ba5930134797bed7dc5e6"
Oct 31 13:45:08.422397 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:08.422372    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4af763bfc4b28c4c6b9ad53de8620dee77c08acb22ba5930134797bed7dc5e6\": container with ID starting with f4af763bfc4b28c4c6b9ad53de8620dee77c08acb22ba5930134797bed7dc5e6 not found: ID does not exist" containerID="f4af763bfc4b28c4c6b9ad53de8620dee77c08acb22ba5930134797bed7dc5e6"
Oct 31 13:45:08.422562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.422537    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4af763bfc4b28c4c6b9ad53de8620dee77c08acb22ba5930134797bed7dc5e6"} err="failed to get container status \"f4af763bfc4b28c4c6b9ad53de8620dee77c08acb22ba5930134797bed7dc5e6\": rpc error: code = NotFound desc = could not find container \"f4af763bfc4b28c4c6b9ad53de8620dee77c08acb22ba5930134797bed7dc5e6\": container with ID starting with f4af763bfc4b28c4c6b9ad53de8620dee77c08acb22ba5930134797bed7dc5e6 not found: ID does not exist"
Oct 31 13:45:08.422715 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.422701    4428 scope.go:122] "RemoveContainer" containerID="765142255aff4fff167f6035389dd098a29fed8cb1425591e6107036ff29c1ee"
Oct 31 13:45:08.440735 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.440709    4428 scope.go:122] "RemoveContainer" containerID="765142255aff4fff167f6035389dd098a29fed8cb1425591e6107036ff29c1ee"
Oct 31 13:45:08.441184 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:08.441157    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765142255aff4fff167f6035389dd098a29fed8cb1425591e6107036ff29c1ee\": container with ID starting with 765142255aff4fff167f6035389dd098a29fed8cb1425591e6107036ff29c1ee not found: ID does not exist" containerID="765142255aff4fff167f6035389dd098a29fed8cb1425591e6107036ff29c1ee"
Oct 31 13:45:08.441335 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.441198    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765142255aff4fff167f6035389dd098a29fed8cb1425591e6107036ff29c1ee"} err="failed to get container status \"765142255aff4fff167f6035389dd098a29fed8cb1425591e6107036ff29c1ee\": rpc error: code = NotFound desc = could not find container \"765142255aff4fff167f6035389dd098a29fed8cb1425591e6107036ff29c1ee\": container with ID starting with 765142255aff4fff167f6035389dd098a29fed8cb1425591e6107036ff29c1ee not found: ID does not exist"
Oct 31 13:45:08.445678 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.445653    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"]
Oct 31 13:45:08.454847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.454807    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d4b52a-69dc48fff5-cmz2p"]
Oct 31 13:45:08.481436 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.481405    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd12e2cb-e569-446e-8cbf-d6cd4779c833-kube-api-access-r9nzx\" (UniqueName: \"kubernetes.io/projected/dd12e2cb-e569-446e-8cbf-d6cd4779c833-kube-api-access-r9nzx\") pod \"dd12e2cb-e569-446e-8cbf-d6cd4779c833\" (UID: \"dd12e2cb-e569-446e-8cbf-d6cd4779c833\") "
Oct 31 13:45:08.481711 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.481694    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd12e2cb-e569-446e-8cbf-d6cd4779c833-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd12e2cb-e569-446e-8cbf-d6cd4779c833-plugins-config-volume\") pod \"dd12e2cb-e569-446e-8cbf-d6cd4779c833\" (UID: \"dd12e2cb-e569-446e-8cbf-d6cd4779c833\") "
Oct 31 13:45:08.482188 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.482155    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd12e2cb-e569-446e-8cbf-d6cd4779c833-plugins-config-volume" pod "dd12e2cb-e569-446e-8cbf-d6cd4779c833" (UID: "dd12e2cb-e569-446e-8cbf-d6cd4779c833"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:08.482452 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.482436    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngcnv\" (UniqueName: \"kubernetes.io/projected/9cf05b3f-e7f7-4b17-b06c-c015ae338bad-kube-api-access-ngcnv\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:08.482537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.482454    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd12e2cb-e569-446e-8cbf-d6cd4779c833-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:08.484452 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.484423    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd12e2cb-e569-446e-8cbf-d6cd4779c833-kube-api-access-r9nzx" pod "dd12e2cb-e569-446e-8cbf-d6cd4779c833" (UID: "dd12e2cb-e569-446e-8cbf-d6cd4779c833"). InnerVolumeSpecName "kube-api-access-r9nzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:08.583645 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.583611    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9nzx\" (UniqueName: \"kubernetes.io/projected/dd12e2cb-e569-446e-8cbf-d6cd4779c833-kube-api-access-r9nzx\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:08.735329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.735296    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"]
Oct 31 13:45:08.743821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.743788    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d4b52a-5f746bd8c9-qqs6s"]
Oct 31 13:45:08.759715 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.759684    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cf05b3f-e7f7-4b17-b06c-c015ae338bad" path="/var/lib/kubelet/pods/9cf05b3f-e7f7-4b17-b06c-c015ae338bad/volumes"
Oct 31 13:45:08.760281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.760264    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd12e2cb-e569-446e-8cbf-d6cd4779c833" path="/var/lib/kubelet/pods/dd12e2cb-e569-446e-8cbf-d6cd4779c833/volumes"
Oct 31 13:45:08.803150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:08.800045    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:09.483685 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:09.479815    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:09.697945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:09.695371    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:09.816677 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:09.814391    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"]
Oct 31 13:45:09.816677 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:09.814676    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q" podUID="db1e5779-e42f-48ed-9804-31af5494ce8a" containerName="istio-proxy" containerID="cri-o://5c8846842f03cd5ac360c832a2cd850d6339161b69503afb5f5fef2850a22242" gracePeriod=30
Oct 31 13:45:09.821420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:09.821295    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"]
Oct 31 13:45:09.822630 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:09.822595    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf" podUID="2ccec53f-f517-420f-b20d-2604a73f2884" containerName="istio-proxy" containerID="cri-o://d9f56ee82b3c5380e7564ee713f0b9b1f5a0beb7a13cec339c1bb030a1449575" gracePeriod=30
Oct 31 13:45:09.872361 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:09.872318    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:10.715369 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:10.715317    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:10.932413 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:10.932378    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:11.209340 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:11.209253    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:11.427893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:11.418622    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:11.442895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:11.429249    4428 patch_prober.go:29] interesting pod/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:45:11.444363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:11.444316    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q" podUID="db1e5779-e42f-48ed-9804-31af5494ce8a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:45:12.327462 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:12.327427    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:13.115353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:13.115315    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:14.122704 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:14.122671    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:14.299796 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:14.299753    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:14.464827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:14.464797    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:15.184342 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.184000    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:45:15.187038 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.187012    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:45:15.301316 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.301267    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ccec53f-f517-420f-b20d-2604a73f2884-kube-api-access-9h86x\" (UniqueName: \"kubernetes.io/projected/2ccec53f-f517-420f-b20d-2604a73f2884-kube-api-access-9h86x\") pod \"2ccec53f-f517-420f-b20d-2604a73f2884\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") "
Oct 31 13:45:15.301529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.301332    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-workload-certs\") pod \"db1e5779-e42f-48ed-9804-31af5494ce8a\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") "
Oct 31 13:45:15.301529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.301406    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-istio-data\") pod \"2ccec53f-f517-420f-b20d-2604a73f2884\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") "
Oct 31 13:45:15.301529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.301435    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-data\") pod \"db1e5779-e42f-48ed-9804-31af5494ce8a\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") "
Oct 31 13:45:15.301529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.301528    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-credential-socket\") pod \"2ccec53f-f517-420f-b20d-2604a73f2884\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") "
Oct 31 13:45:15.301759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.301667    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-workload-certs" pod "db1e5779-e42f-48ed-9804-31af5494ce8a" (UID: "db1e5779-e42f-48ed-9804-31af5494ce8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:15.301822 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.301782    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-istio-data" pod "2ccec53f-f517-420f-b20d-2604a73f2884" (UID: "2ccec53f-f517-420f-b20d-2604a73f2884"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:15.301822 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.301800    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-data" pod "db1e5779-e42f-48ed-9804-31af5494ce8a" (UID: "db1e5779-e42f-48ed-9804-31af5494ce8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:15.302023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.302003    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ccec53f-f517-420f-b20d-2604a73f2884-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ccec53f-f517-420f-b20d-2604a73f2884-istio-podinfo\") pod \"2ccec53f-f517-420f-b20d-2604a73f2884\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") "
Oct 31 13:45:15.302105 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.302013    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-credential-socket" pod "2ccec53f-f517-420f-b20d-2604a73f2884" (UID: "2ccec53f-f517-420f-b20d-2604a73f2884"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:15.302105 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.302062    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ccec53f-f517-420f-b20d-2604a73f2884-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ccec53f-f517-420f-b20d-2604a73f2884-istiod-ca-cert\") pod \"2ccec53f-f517-420f-b20d-2604a73f2884\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") "
Oct 31 13:45:15.302219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.302140    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ccec53f-f517-420f-b20d-2604a73f2884-istio-token\" (UniqueName: \"kubernetes.io/projected/2ccec53f-f517-420f-b20d-2604a73f2884-istio-token\") pod \"2ccec53f-f517-420f-b20d-2604a73f2884\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") "
Oct 31 13:45:15.302219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.302179    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-podinfo\") pod \"db1e5779-e42f-48ed-9804-31af5494ce8a\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") "
Oct 31 13:45:15.302334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.302218    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-istio-envoy\") pod \"2ccec53f-f517-420f-b20d-2604a73f2884\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") "
Oct 31 13:45:15.302334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.302313    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-workload-certs\") pod \"2ccec53f-f517-420f-b20d-2604a73f2884\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") "
Oct 31 13:45:15.302449 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.302378    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-workload-socket\") pod \"db1e5779-e42f-48ed-9804-31af5494ce8a\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") "
Oct 31 13:45:15.302449 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.302405    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-envoy\") pod \"db1e5779-e42f-48ed-9804-31af5494ce8a\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") "
Oct 31 13:45:15.302557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.302516    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-token\" (UniqueName: \"kubernetes.io/projected/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-token\") pod \"db1e5779-e42f-48ed-9804-31af5494ce8a\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") "
Oct 31 13:45:15.302618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.302558    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1e5779-e42f-48ed-9804-31af5494ce8a-kube-api-access-cq6g4\" (UniqueName: \"kubernetes.io/projected/db1e5779-e42f-48ed-9804-31af5494ce8a-kube-api-access-cq6g4\") pod \"db1e5779-e42f-48ed-9804-31af5494ce8a\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") "
Oct 31 13:45:15.302618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.302594    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-credential-socket\") pod \"db1e5779-e42f-48ed-9804-31af5494ce8a\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") "
Oct 31 13:45:15.302725 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.302623    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db1e5779-e42f-48ed-9804-31af5494ce8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1e5779-e42f-48ed-9804-31af5494ce8a-istiod-ca-cert\") pod \"db1e5779-e42f-48ed-9804-31af5494ce8a\" (UID: \"db1e5779-e42f-48ed-9804-31af5494ce8a\") "
Oct 31 13:45:15.302725 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.302621    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ccec53f-f517-420f-b20d-2604a73f2884-istiod-ca-cert" pod "2ccec53f-f517-420f-b20d-2604a73f2884" (UID: "2ccec53f-f517-420f-b20d-2604a73f2884"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:15.302835 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.302738    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-workload-socket\") pod \"2ccec53f-f517-420f-b20d-2604a73f2884\" (UID: \"2ccec53f-f517-420f-b20d-2604a73f2884\") "
Oct 31 13:45:15.302963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.302938    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-workload-socket" pod "db1e5779-e42f-48ed-9804-31af5494ce8a" (UID: "db1e5779-e42f-48ed-9804-31af5494ce8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:15.303836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.303805    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.303998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.303839    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.303998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.303854    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.303998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.303886    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.303998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.303886    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-credential-socket" pod "db1e5779-e42f-48ed-9804-31af5494ce8a" (UID: "db1e5779-e42f-48ed-9804-31af5494ce8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:15.303998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.303900    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ccec53f-f517-420f-b20d-2604a73f2884-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.303998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.303915    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.304518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.304148    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-workload-socket" pod "2ccec53f-f517-420f-b20d-2604a73f2884" (UID: "2ccec53f-f517-420f-b20d-2604a73f2884"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:15.304518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.304432    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1e5779-e42f-48ed-9804-31af5494ce8a-istiod-ca-cert" pod "db1e5779-e42f-48ed-9804-31af5494ce8a" (UID: "db1e5779-e42f-48ed-9804-31af5494ce8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:15.305676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.305643    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ccec53f-f517-420f-b20d-2604a73f2884-kube-api-access-9h86x" pod "2ccec53f-f517-420f-b20d-2604a73f2884" (UID: "2ccec53f-f517-420f-b20d-2604a73f2884"). InnerVolumeSpecName "kube-api-access-9h86x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:15.306030 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.305979    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-workload-certs" pod "2ccec53f-f517-420f-b20d-2604a73f2884" (UID: "2ccec53f-f517-420f-b20d-2604a73f2884"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:15.306506 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.306475    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ccec53f-f517-420f-b20d-2604a73f2884-istio-token" pod "2ccec53f-f517-420f-b20d-2604a73f2884" (UID: "2ccec53f-f517-420f-b20d-2604a73f2884"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:15.307142 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.307115    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-istio-envoy" pod "2ccec53f-f517-420f-b20d-2604a73f2884" (UID: "2ccec53f-f517-420f-b20d-2604a73f2884"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:15.307256 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.307141    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-token" pod "db1e5779-e42f-48ed-9804-31af5494ce8a" (UID: "db1e5779-e42f-48ed-9804-31af5494ce8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:15.307894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.307856    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ccec53f-f517-420f-b20d-2604a73f2884-istio-podinfo" pod "2ccec53f-f517-420f-b20d-2604a73f2884" (UID: "2ccec53f-f517-420f-b20d-2604a73f2884"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:45:15.307990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.307965    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-podinfo" pod "db1e5779-e42f-48ed-9804-31af5494ce8a" (UID: "db1e5779-e42f-48ed-9804-31af5494ce8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:45:15.308238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.308209    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1e5779-e42f-48ed-9804-31af5494ce8a-kube-api-access-cq6g4" pod "db1e5779-e42f-48ed-9804-31af5494ce8a" (UID: "db1e5779-e42f-48ed-9804-31af5494ce8a"). InnerVolumeSpecName "kube-api-access-cq6g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:15.308803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.308779    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-envoy" pod "db1e5779-e42f-48ed-9804-31af5494ce8a" (UID: "db1e5779-e42f-48ed-9804-31af5494ce8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:15.386611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.386505    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:15.406200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.406169    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h86x\" (UniqueName: \"kubernetes.io/projected/2ccec53f-f517-420f-b20d-2604a73f2884-kube-api-access-9h86x\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.406420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.406198    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ccec53f-f517-420f-b20d-2604a73f2884-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.406420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.406233    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ccec53f-f517-420f-b20d-2604a73f2884-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.406420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.406242    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.406420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.406251    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.406420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.406260    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.406420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.406270    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.406420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.406283    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1e5779-e42f-48ed-9804-31af5494ce8a-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.406420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.406296    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq6g4\" (UniqueName: \"kubernetes.io/projected/db1e5779-e42f-48ed-9804-31af5494ce8a-kube-api-access-cq6g4\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.406420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.406350    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1e5779-e42f-48ed-9804-31af5494ce8a-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.406420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.406365    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1e5779-e42f-48ed-9804-31af5494ce8a-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.406420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.406378    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccec53f-f517-420f-b20d-2604a73f2884-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:15.592894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.590272    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:15.776895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.775542    4428 generic.go:356] "Generic (PLEG): container finished" podID="2ccec53f-f517-420f-b20d-2604a73f2884" containerID="d9f56ee82b3c5380e7564ee713f0b9b1f5a0beb7a13cec339c1bb030a1449575" exitCode=0
Oct 31 13:45:15.776895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.775764    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf" event={"ID":"2ccec53f-f517-420f-b20d-2604a73f2884","Type":"ContainerDied","Data":"d9f56ee82b3c5380e7564ee713f0b9b1f5a0beb7a13cec339c1bb030a1449575"}
Oct 31 13:45:15.776895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.775821    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf" event={"ID":"2ccec53f-f517-420f-b20d-2604a73f2884","Type":"ContainerDied","Data":"11262cf109eeed8cd66e3498cecc13ffc0241616e5a3a5687b5de954e9776c12"}
Oct 31 13:45:15.776895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.775851    4428 scope.go:122] "RemoveContainer" containerID="d9f56ee82b3c5380e7564ee713f0b9b1f5a0beb7a13cec339c1bb030a1449575"
Oct 31 13:45:15.793325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.793295    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"
Oct 31 13:45:15.802984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.796214    4428 generic.go:356] "Generic (PLEG): container finished" podID="db1e5779-e42f-48ed-9804-31af5494ce8a" containerID="5c8846842f03cd5ac360c832a2cd850d6339161b69503afb5f5fef2850a22242" exitCode=0
Oct 31 13:45:15.802984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.796389    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q" event={"ID":"db1e5779-e42f-48ed-9804-31af5494ce8a","Type":"ContainerDied","Data":"5c8846842f03cd5ac360c832a2cd850d6339161b69503afb5f5fef2850a22242"}
Oct 31 13:45:15.802984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.796441    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q" event={"ID":"db1e5779-e42f-48ed-9804-31af5494ce8a","Type":"ContainerDied","Data":"a692a5dc099d4126d0fbe8387c8c3eaf6e865c1270522cb0b5fbf2b25256f7c5"}
Oct 31 13:45:15.802984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.796704    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"
Oct 31 13:45:15.807824 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.806718    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:15.820125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.815019    4428 scope.go:122] "RemoveContainer" containerID="fddf8f8bc7bb0e249ae0e4d6dbd84845abd85605e4ca349ffb8d374bba279f5a"
Oct 31 13:45:15.910678 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.910640    4428 scope.go:122] "RemoveContainer" containerID="d9f56ee82b3c5380e7564ee713f0b9b1f5a0beb7a13cec339c1bb030a1449575"
Oct 31 13:45:15.911376 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:15.911331    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9f56ee82b3c5380e7564ee713f0b9b1f5a0beb7a13cec339c1bb030a1449575\": container with ID starting with d9f56ee82b3c5380e7564ee713f0b9b1f5a0beb7a13cec339c1bb030a1449575 not found: ID does not exist" containerID="d9f56ee82b3c5380e7564ee713f0b9b1f5a0beb7a13cec339c1bb030a1449575"
Oct 31 13:45:15.911503 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.911378    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f56ee82b3c5380e7564ee713f0b9b1f5a0beb7a13cec339c1bb030a1449575"} err="failed to get container status \"d9f56ee82b3c5380e7564ee713f0b9b1f5a0beb7a13cec339c1bb030a1449575\": rpc error: code = NotFound desc = could not find container \"d9f56ee82b3c5380e7564ee713f0b9b1f5a0beb7a13cec339c1bb030a1449575\": container with ID starting with d9f56ee82b3c5380e7564ee713f0b9b1f5a0beb7a13cec339c1bb030a1449575 not found: ID does not exist"
Oct 31 13:45:15.911503 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.911407    4428 scope.go:122] "RemoveContainer" containerID="fddf8f8bc7bb0e249ae0e4d6dbd84845abd85605e4ca349ffb8d374bba279f5a"
Oct 31 13:45:15.911988 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:15.911961    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fddf8f8bc7bb0e249ae0e4d6dbd84845abd85605e4ca349ffb8d374bba279f5a\": container with ID starting with fddf8f8bc7bb0e249ae0e4d6dbd84845abd85605e4ca349ffb8d374bba279f5a not found: ID does not exist" containerID="fddf8f8bc7bb0e249ae0e4d6dbd84845abd85605e4ca349ffb8d374bba279f5a"
Oct 31 13:45:15.912092 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.911997    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fddf8f8bc7bb0e249ae0e4d6dbd84845abd85605e4ca349ffb8d374bba279f5a"} err="failed to get container status \"fddf8f8bc7bb0e249ae0e4d6dbd84845abd85605e4ca349ffb8d374bba279f5a\": rpc error: code = NotFound desc = could not find container \"fddf8f8bc7bb0e249ae0e4d6dbd84845abd85605e4ca349ffb8d374bba279f5a\": container with ID starting with fddf8f8bc7bb0e249ae0e4d6dbd84845abd85605e4ca349ffb8d374bba279f5a not found: ID does not exist"
Oct 31 13:45:15.912092 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.912027    4428 scope.go:122] "RemoveContainer" containerID="5c8846842f03cd5ac360c832a2cd850d6339161b69503afb5f5fef2850a22242"
Oct 31 13:45:15.930051 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.930023    4428 scope.go:122] "RemoveContainer" containerID="5c8846842f03cd5ac360c832a2cd850d6339161b69503afb5f5fef2850a22242"
Oct 31 13:45:15.930426 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:15.930404    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8846842f03cd5ac360c832a2cd850d6339161b69503afb5f5fef2850a22242\": container with ID starting with 5c8846842f03cd5ac360c832a2cd850d6339161b69503afb5f5fef2850a22242 not found: ID does not exist" containerID="5c8846842f03cd5ac360c832a2cd850d6339161b69503afb5f5fef2850a22242"
Oct 31 13:45:15.930494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.930436    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8846842f03cd5ac360c832a2cd850d6339161b69503afb5f5fef2850a22242"} err="failed to get container status \"5c8846842f03cd5ac360c832a2cd850d6339161b69503afb5f5fef2850a22242\": rpc error: code = NotFound desc = could not find container \"5c8846842f03cd5ac360c832a2cd850d6339161b69503afb5f5fef2850a22242\": container with ID starting with 5c8846842f03cd5ac360c832a2cd850d6339161b69503afb5f5fef2850a22242 not found: ID does not exist"
Oct 31 13:45:15.954466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.954438    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"]
Oct 31 13:45:15.984447 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:15.984416    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-l6g6q"]
Oct 31 13:45:16.022619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.022584    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"]
Oct 31 13:45:16.029035 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.028963    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d4b52a-openshift-default-7d99d6d877-qpnrf"]
Oct 31 13:45:16.707009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.706962    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:16.729041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.729005    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"]
Oct 31 13:45:16.730114 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.730095    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9cf05b3f-e7f7-4b17-b06c-c015ae338bad" containerName="payload-processing"
Oct 31 13:45:16.730114 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.730115    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cf05b3f-e7f7-4b17-b06c-c015ae338bad" containerName="payload-processing"
Oct 31 13:45:16.730253 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.730153    4428 cpu_manager.go:414] "containerMap: removing container" podUID="2ccec53f-f517-420f-b20d-2604a73f2884" containerName="istio-proxy"
Oct 31 13:45:16.730253 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.730158    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ccec53f-f517-420f-b20d-2604a73f2884" containerName="istio-proxy"
Oct 31 13:45:16.730253 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.730176    4428 cpu_manager.go:414] "containerMap: removing container" podUID="dd12e2cb-e569-446e-8cbf-d6cd4779c833" containerName="payload-pre-processing"
Oct 31 13:45:16.730253 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.730181    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd12e2cb-e569-446e-8cbf-d6cd4779c833" containerName="payload-pre-processing"
Oct 31 13:45:16.730253 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.730202    4428 cpu_manager.go:414] "containerMap: removing container" podUID="2ccec53f-f517-420f-b20d-2604a73f2884" containerName="istio-proxy"
Oct 31 13:45:16.730253 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.730206    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ccec53f-f517-420f-b20d-2604a73f2884" containerName="istio-proxy"
Oct 31 13:45:16.730253 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.730220    4428 cpu_manager.go:414] "containerMap: removing container" podUID="db1e5779-e42f-48ed-9804-31af5494ce8a" containerName="istio-proxy"
Oct 31 13:45:16.730253 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.730225    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1e5779-e42f-48ed-9804-31af5494ce8a" containerName="istio-proxy"
Oct 31 13:45:16.730888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.730346    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd12e2cb-e569-446e-8cbf-d6cd4779c833" containerName="payload-pre-processing"
Oct 31 13:45:16.730888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.730357    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ccec53f-f517-420f-b20d-2604a73f2884" containerName="istio-proxy"
Oct 31 13:45:16.730888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.730364    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1e5779-e42f-48ed-9804-31af5494ce8a" containerName="istio-proxy"
Oct 31 13:45:16.730888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.730373    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cf05b3f-e7f7-4b17-b06c-c015ae338bad" containerName="payload-processing"
Oct 31 13:45:16.736056 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.736037    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"
Oct 31 13:45:16.751017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.750988    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-6a3533\""
Oct 31 13:45:16.762359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.762302    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ccec53f-f517-420f-b20d-2604a73f2884" path="/var/lib/kubelet/pods/2ccec53f-f517-420f-b20d-2604a73f2884/volumes"
Oct 31 13:45:16.763292 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.763172    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db1e5779-e42f-48ed-9804-31af5494ce8a" path="/var/lib/kubelet/pods/db1e5779-e42f-48ed-9804-31af5494ce8a/volumes"
Oct 31 13:45:16.763700 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.763672    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"]
Oct 31 13:45:16.774853 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.772466    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"]
Oct 31 13:45:16.774853 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.774119    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ccec53f-f517-420f-b20d-2604a73f2884" containerName="istio-proxy"
Oct 31 13:45:16.780318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.780291    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"
Oct 31 13:45:16.794936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.789908    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"]
Oct 31 13:45:16.794936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.792035    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-6a3533-dockercfg-pwd9b\""
Oct 31 13:45:16.923896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.910297    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ee4cb2-a701-44af-a9d3-538ef85c939c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj\" (UID: \"67ee4cb2-a701-44af-a9d3-538ef85c939c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"
Oct 31 13:45:16.923896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.910451    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c622d692-1344-4482-9c4c-c26ae52a6642-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96\" (UID: \"c622d692-1344-4482-9c4c-c26ae52a6642\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"
Oct 31 13:45:16.923896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.910538    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztfvd\" (UniqueName: \"kubernetes.io/projected/67ee4cb2-a701-44af-a9d3-538ef85c939c-kube-api-access-ztfvd\") pod \"payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj\" (UID: \"67ee4cb2-a701-44af-a9d3-538ef85c939c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"
Oct 31 13:45:16.923896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.910679    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7pf9\" (UniqueName: \"kubernetes.io/projected/c622d692-1344-4482-9c4c-c26ae52a6642-kube-api-access-r7pf9\") pod \"payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96\" (UID: \"c622d692-1344-4482-9c4c-c26ae52a6642\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"
Oct 31 13:45:16.937906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:16.936471    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:17.014924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.014831    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7pf9\" (UniqueName: \"kubernetes.io/projected/c622d692-1344-4482-9c4c-c26ae52a6642-kube-api-access-r7pf9\") pod \"payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96\" (UID: \"c622d692-1344-4482-9c4c-c26ae52a6642\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"
Oct 31 13:45:17.015384 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.015356    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ee4cb2-a701-44af-a9d3-538ef85c939c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj\" (UID: \"67ee4cb2-a701-44af-a9d3-538ef85c939c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"
Oct 31 13:45:17.015614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.015597    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c622d692-1344-4482-9c4c-c26ae52a6642-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96\" (UID: \"c622d692-1344-4482-9c4c-c26ae52a6642\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"
Oct 31 13:45:17.015771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.015757    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztfvd\" (UniqueName: \"kubernetes.io/projected/67ee4cb2-a701-44af-a9d3-538ef85c939c-kube-api-access-ztfvd\") pod \"payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj\" (UID: \"67ee4cb2-a701-44af-a9d3-538ef85c939c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"
Oct 31 13:45:17.031893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.023015    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c622d692-1344-4482-9c4c-c26ae52a6642-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96\" (UID: \"c622d692-1344-4482-9c4c-c26ae52a6642\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"
Oct 31 13:45:17.033908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.033859    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ee4cb2-a701-44af-a9d3-538ef85c939c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj\" (UID: \"67ee4cb2-a701-44af-a9d3-538ef85c939c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"
Oct 31 13:45:17.102892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.102655    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztfvd\" (UniqueName: \"kubernetes.io/projected/67ee4cb2-a701-44af-a9d3-538ef85c939c-kube-api-access-ztfvd\") pod \"payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj\" (UID: \"67ee4cb2-a701-44af-a9d3-538ef85c939c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"
Oct 31 13:45:17.107899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.103337    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7pf9\" (UniqueName: \"kubernetes.io/projected/c622d692-1344-4482-9c4c-c26ae52a6642-kube-api-access-r7pf9\") pod \"payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96\" (UID: \"c622d692-1344-4482-9c4c-c26ae52a6642\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"
Oct 31 13:45:17.139895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.132717    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"
Oct 31 13:45:17.353006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.352972    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"
Oct 31 13:45:17.524258 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.524228    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"]
Oct 31 13:45:17.526147 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:45:17.526119    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc622d692_1344_4482_9c4c_c26ae52a6642.slice/crio-b861e9570d13009858b9686f6d7bd3664ab88c2df542d84e6fb30f55de6932d6 WatchSource:0}: Error finding container b861e9570d13009858b9686f6d7bd3664ab88c2df542d84e6fb30f55de6932d6: Status 404 returned error can't find the container with id b861e9570d13009858b9686f6d7bd3664ab88c2df542d84e6fb30f55de6932d6
Oct 31 13:45:17.666488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.666449    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"]
Oct 31 13:45:17.666602 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:45:17.666581    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ee4cb2_a701_44af_a9d3_538ef85c939c.slice/crio-97e38df8bdb99f9cc4a743321a00ce9aadd910b5697e53d62f39bfb2bce208ce WatchSource:0}: Error finding container 97e38df8bdb99f9cc4a743321a00ce9aadd910b5697e53d62f39bfb2bce208ce: Status 404 returned error can't find the container with id 97e38df8bdb99f9cc4a743321a00ce9aadd910b5697e53d62f39bfb2bce208ce
Oct 31 13:45:17.747838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.747808    4428 scope.go:122] "RemoveContainer" containerID="40ee909b2cb40195dcfcb96dbcf4f9284db502e2522c2e6a01135c9706904ddf"
Oct 31 13:45:17.748300 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.748267    4428 scope.go:122] "RemoveContainer" containerID="defb4dc60bf40c064b39850a66f6f0eec5aff01a482580c0e2c81137a4b1d116"
Oct 31 13:45:17.748383 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.748339    4428 scope.go:122] "RemoveContainer" containerID="731aa92bbb49c5ebd93e4b7a8ca5b8ce99fd8e48fc09b76f481dd0feb5a957e3"
Oct 31 13:45:17.748604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.748581    4428 scope.go:122] "RemoveContainer" containerID="f191504f3d381562535e6a2cb2e90ba2f9769a15907ec81e4c6d47ca120f99d3"
Oct 31 13:45:17.828307 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.828275    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96" event={"ID":"c622d692-1344-4482-9c4c-c26ae52a6642","Type":"ContainerStarted","Data":"8c5defb178a8e6c27c18b9978d94f82850c367db196518a2a2f102b132004c9e"}
Oct 31 13:45:17.828496 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.828345    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96" event={"ID":"c622d692-1344-4482-9c4c-c26ae52a6642","Type":"ContainerStarted","Data":"b861e9570d13009858b9686f6d7bd3664ab88c2df542d84e6fb30f55de6932d6"}
Oct 31 13:45:17.828496 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.828474    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"
Oct 31 13:45:17.831129 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.831100    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj" event={"ID":"67ee4cb2-a701-44af-a9d3-538ef85c939c","Type":"ContainerStarted","Data":"1413b9af9952455389618bcd762edca94c7c64246f3dae1aa8407cee48686dcb"}
Oct 31 13:45:17.831222 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.831157    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj" event={"ID":"67ee4cb2-a701-44af-a9d3-538ef85c939c","Type":"ContainerStarted","Data":"97e38df8bdb99f9cc4a743321a00ce9aadd910b5697e53d62f39bfb2bce208ce"}
Oct 31 13:45:17.831438 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.831421    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"
Oct 31 13:45:17.850454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.850393    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96" podStartSLOduration=1.8503745600000001 podStartE2EDuration="1.85037456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:17.847340313 +0000 UTC m=+2515.245688938" watchObservedRunningTime="2026-10-31 13:45:17.85037456 +0000 UTC m=+2515.248723187"
Oct 31 13:45:17.874033 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:17.873980    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj" podStartSLOduration=1.873966893 podStartE2EDuration="1.873966893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:17.868904647 +0000 UTC m=+2515.267253264" watchObservedRunningTime="2026-10-31 13:45:17.873966893 +0000 UTC m=+2515.272315516"
Oct 31 13:45:18.122124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:18.122092    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:18.744046 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:18.743995    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:45:18.748306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:18.748275    4428 scope.go:122] "RemoveContainer" containerID="07229caea95318bbdfc954a269a093763ffcfce6dbffa44422cbe87775b61d8a"
Oct 31 13:45:18.749067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:18.749033    4428 scope.go:122] "RemoveContainer" containerID="1edc619b8b375f711577f4b1b0ac0e588cc1908aa39a2e30ddf64279c99b2091"
Oct 31 13:45:18.749560 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:18.749536    4428 scope.go:122] "RemoveContainer" containerID="ff473a4d21cc30d7cd596ca6c96bd79c6d39b0ecaf4690c570084fbb3516235f"
Oct 31 13:45:18.749747 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:18.749719    4428 scope.go:122] "RemoveContainer" containerID="d3908faeff05e0b906b31adfea188260f2250418efad81f9d9a7e2b0e21dba40"
Oct 31 13:45:18.750149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:18.750127    4428 scope.go:122] "RemoveContainer" containerID="9fa384df2d11aebb2fba2360adc408bf7f9aa7767046215aceb69aa61e821542"
Oct 31 13:45:18.841809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:18.841773    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67" event={"ID":"61fe9f57-eaa6-49c1-9a63-f37485554a23","Type":"ContainerStarted","Data":"38e4374d4fc7e0bf00aadb4f7ee5d5a2281a49d45b18e9e3bf0706f13f089bbb"}
Oct 31 13:45:18.844461 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:18.844431    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp" event={"ID":"9daa213c-e1ea-473d-8ed6-a2bc15630b58","Type":"ContainerStarted","Data":"a952a21130269853de2d0afe5cf9c89c11263fe17b0ee8bfd153ecd8bedcfdf6"}
Oct 31 13:45:18.848143 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:18.848111    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw" event={"ID":"dc233cb4-c11a-4bd5-9abb-951884813235","Type":"ContainerStarted","Data":"787bb9c84d3091c6300fe9ebfda8f3f2ebd74a5b7897b527a879a7a36b693b43"}
Oct 31 13:45:18.851927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:18.851719    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76" event={"ID":"d507266c-1d74-4d2c-9af8-c9990fe98174","Type":"ContainerStarted","Data":"24ad68245cb5e1dc57e40bfbbab55fc9fdb775a855f762b357953b4e854df412"}
Oct 31 13:45:19.075997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.054481    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:19.748975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.748944    4428 scope.go:122] "RemoveContainer" containerID="72afb27e590373ac7f90f3bc05927562a4b4661a9b07a93920bd9997fe37c26d"
Oct 31 13:45:19.865100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.865070    4428 generic.go:356] "Generic (PLEG): container finished" podID="61fe9f57-eaa6-49c1-9a63-f37485554a23" containerID="38e4374d4fc7e0bf00aadb4f7ee5d5a2281a49d45b18e9e3bf0706f13f089bbb" exitCode=60
Oct 31 13:45:19.865235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.865143    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67" event={"ID":"61fe9f57-eaa6-49c1-9a63-f37485554a23","Type":"ContainerDied","Data":"38e4374d4fc7e0bf00aadb4f7ee5d5a2281a49d45b18e9e3bf0706f13f089bbb"}
Oct 31 13:45:19.865235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.865202    4428 scope.go:122] "RemoveContainer" containerID="f191504f3d381562535e6a2cb2e90ba2f9769a15907ec81e4c6d47ca120f99d3"
Oct 31 13:45:19.865702 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.865682    4428 scope.go:122] "RemoveContainer" containerID="38e4374d4fc7e0bf00aadb4f7ee5d5a2281a49d45b18e9e3bf0706f13f089bbb"
Oct 31 13:45:19.866259 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:19.866147    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67_odh-ai-gateway-infra(61fe9f57-eaa6-49c1-9a63-f37485554a23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67" podUID="61fe9f57-eaa6-49c1-9a63-f37485554a23"
Oct 31 13:45:19.867470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.867441    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4" event={"ID":"9485106e-58f9-4b42-ae34-0136f6345af2","Type":"ContainerStarted","Data":"49f403cfdba672172e5c15f6d6b85daa8f5c2983187892f34e50034c8cf44df6"}
Oct 31 13:45:19.871660 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.871637    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8" event={"ID":"6bb5549e-6008-46a8-8e55-3b37cb62cd60","Type":"ContainerStarted","Data":"27a4efe978c1db6ed5aed81873b84e475064e1f86a1b31c68b53b678f14ebf14"}
Oct 31 13:45:19.873914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.873886    4428 generic.go:356] "Generic (PLEG): container finished" podID="9daa213c-e1ea-473d-8ed6-a2bc15630b58" containerID="a952a21130269853de2d0afe5cf9c89c11263fe17b0ee8bfd153ecd8bedcfdf6" exitCode=60
Oct 31 13:45:19.873914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.873898    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp" event={"ID":"9daa213c-e1ea-473d-8ed6-a2bc15630b58","Type":"ContainerDied","Data":"a952a21130269853de2d0afe5cf9c89c11263fe17b0ee8bfd153ecd8bedcfdf6"}
Oct 31 13:45:19.874235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.874218    4428 scope.go:122] "RemoveContainer" containerID="a952a21130269853de2d0afe5cf9c89c11263fe17b0ee8bfd153ecd8bedcfdf6"
Oct 31 13:45:19.874506 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:19.874485    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp_odh-ai-gateway-infra(9daa213c-e1ea-473d-8ed6-a2bc15630b58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp" podUID="9daa213c-e1ea-473d-8ed6-a2bc15630b58"
Oct 31 13:45:19.876181 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.876164    4428 generic.go:356] "Generic (PLEG): container finished" podID="dc233cb4-c11a-4bd5-9abb-951884813235" containerID="787bb9c84d3091c6300fe9ebfda8f3f2ebd74a5b7897b527a879a7a36b693b43" exitCode=60
Oct 31 13:45:19.876273 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.876240    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw" event={"ID":"dc233cb4-c11a-4bd5-9abb-951884813235","Type":"ContainerDied","Data":"787bb9c84d3091c6300fe9ebfda8f3f2ebd74a5b7897b527a879a7a36b693b43"}
Oct 31 13:45:19.876981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.876699    4428 scope.go:122] "RemoveContainer" containerID="787bb9c84d3091c6300fe9ebfda8f3f2ebd74a5b7897b527a879a7a36b693b43"
Oct 31 13:45:19.877224 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:19.877192    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw_odh-ai-gateway-infra(dc233cb4-c11a-4bd5-9abb-951884813235)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw" podUID="dc233cb4-c11a-4bd5-9abb-951884813235"
Oct 31 13:45:19.879550 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.879526    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs" event={"ID":"77411480-49b1-49e5-90f0-1fc4416288da","Type":"ContainerStarted","Data":"2d4ccfccd1fc1fed31c22081907201ff4e6ae524ec8707652eac01274e109078"}
Oct 31 13:45:19.881618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.881599    4428 generic.go:356] "Generic (PLEG): container finished" podID="d507266c-1d74-4d2c-9af8-c9990fe98174" containerID="24ad68245cb5e1dc57e40bfbbab55fc9fdb775a855f762b357953b4e854df412" exitCode=60
Oct 31 13:45:19.881719 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.881672    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76" event={"ID":"d507266c-1d74-4d2c-9af8-c9990fe98174","Type":"ContainerDied","Data":"24ad68245cb5e1dc57e40bfbbab55fc9fdb775a855f762b357953b4e854df412"}
Oct 31 13:45:19.881990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.881975    4428 scope.go:122] "RemoveContainer" containerID="24ad68245cb5e1dc57e40bfbbab55fc9fdb775a855f762b357953b4e854df412"
Oct 31 13:45:19.882263 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:19.882238    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76_odh-ai-gateway-infra(d507266c-1d74-4d2c-9af8-c9990fe98174)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76" podUID="d507266c-1d74-4d2c-9af8-c9990fe98174"
Oct 31 13:45:19.882583 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.882564    4428 scope.go:122] "RemoveContainer" containerID="defb4dc60bf40c064b39850a66f6f0eec5aff01a482580c0e2c81137a4b1d116"
Oct 31 13:45:19.883785 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.883762    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp" event={"ID":"680bd885-b3c7-42f5-a3a8-4f2950d0b462","Type":"ContainerStarted","Data":"e5498ae5122afdb74b036f395f0039b1c7203a8a89451ec4476ee0dc8af9610a"}
Oct 31 13:45:19.884327 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.884308    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67"]
Oct 31 13:45:19.898561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.898544    4428 scope.go:122] "RemoveContainer" containerID="731aa92bbb49c5ebd93e4b7a8ca5b8ce99fd8e48fc09b76f481dd0feb5a957e3"
Oct 31 13:45:19.910157 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.910095    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw"]
Oct 31 13:45:19.914525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.914492    4428 scope.go:122] "RemoveContainer" containerID="40ee909b2cb40195dcfcb96dbcf4f9284db502e2522c2e6a01135c9706904ddf"
Oct 31 13:45:19.944342 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.944316    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76"]
Oct 31 13:45:19.957950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:19.957927    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp"]
Oct 31 13:45:20.011740 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.011684    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8" podStartSLOduration=20.011669983 podStartE2EDuration="20.011669983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:20.009617839 +0000 UTC m=+2517.407966463" watchObservedRunningTime="2026-10-31 13:45:20.011669983 +0000 UTC m=+2517.410018608"
Oct 31 13:45:20.057472 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.057414    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp" podStartSLOduration=20.057400355 podStartE2EDuration="20.057400355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:20.044891263 +0000 UTC m=+2517.443239881" watchObservedRunningTime="2026-10-31 13:45:20.057400355 +0000 UTC m=+2517.455748979"
Oct 31 13:45:20.207462 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.207355    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:20.896845 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.896744    4428 generic.go:356] "Generic (PLEG): container finished" podID="9485106e-58f9-4b42-ae34-0136f6345af2" containerID="49f403cfdba672172e5c15f6d6b85daa8f5c2983187892f34e50034c8cf44df6" exitCode=60
Oct 31 13:45:20.896845 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.896828    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4" event={"ID":"9485106e-58f9-4b42-ae34-0136f6345af2","Type":"ContainerDied","Data":"49f403cfdba672172e5c15f6d6b85daa8f5c2983187892f34e50034c8cf44df6"}
Oct 31 13:45:20.897456 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.896943    4428 scope.go:122] "RemoveContainer" containerID="1edc619b8b375f711577f4b1b0ac0e588cc1908aa39a2e30ddf64279c99b2091"
Oct 31 13:45:20.897456 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.897440    4428 scope.go:122] "RemoveContainer" containerID="49f403cfdba672172e5c15f6d6b85daa8f5c2983187892f34e50034c8cf44df6"
Oct 31 13:45:20.897955 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:20.897929    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4_odh-ai-gateway-infra(9485106e-58f9-4b42-ae34-0136f6345af2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4" podUID="9485106e-58f9-4b42-ae34-0136f6345af2"
Oct 31 13:45:20.899584 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.899559    4428 generic.go:356] "Generic (PLEG): container finished" podID="55d9f9bc-22ec-4943-8a8c-9ed7abe7af98" containerID="51ee443bec390fd6bba8e12aa3e9e65b6e6a8bb08994333dede7d58ed9aaca86" exitCode=60
Oct 31 13:45:20.899711 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.899614    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8" event={"ID":"55d9f9bc-22ec-4943-8a8c-9ed7abe7af98","Type":"ContainerDied","Data":"51ee443bec390fd6bba8e12aa3e9e65b6e6a8bb08994333dede7d58ed9aaca86"}
Oct 31 13:45:20.900068 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.900050    4428 scope.go:122] "RemoveContainer" containerID="51ee443bec390fd6bba8e12aa3e9e65b6e6a8bb08994333dede7d58ed9aaca86"
Oct 31 13:45:20.900446 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:20.900419    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8_odh-ai-gateway-infra(55d9f9bc-22ec-4943-8a8c-9ed7abe7af98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8" podUID="55d9f9bc-22ec-4943-8a8c-9ed7abe7af98"
Oct 31 13:45:20.901781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.901762    4428 generic.go:356] "Generic (PLEG): container finished" podID="6bb5549e-6008-46a8-8e55-3b37cb62cd60" containerID="27a4efe978c1db6ed5aed81873b84e475064e1f86a1b31c68b53b678f14ebf14" exitCode=60
Oct 31 13:45:20.901861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.901825    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8" event={"ID":"6bb5549e-6008-46a8-8e55-3b37cb62cd60","Type":"ContainerDied","Data":"27a4efe978c1db6ed5aed81873b84e475064e1f86a1b31c68b53b678f14ebf14"}
Oct 31 13:45:20.902236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.902219    4428 scope.go:122] "RemoveContainer" containerID="27a4efe978c1db6ed5aed81873b84e475064e1f86a1b31c68b53b678f14ebf14"
Oct 31 13:45:20.902607 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:20.902578    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8_odh-ai-gateway-infra(6bb5549e-6008-46a8-8e55-3b37cb62cd60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8" podUID="6bb5549e-6008-46a8-8e55-3b37cb62cd60"
Oct 31 13:45:20.908299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.908271    4428 generic.go:356] "Generic (PLEG): container finished" podID="77411480-49b1-49e5-90f0-1fc4416288da" containerID="2d4ccfccd1fc1fed31c22081907201ff4e6ae524ec8707652eac01274e109078" exitCode=60
Oct 31 13:45:20.908391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.908343    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs" event={"ID":"77411480-49b1-49e5-90f0-1fc4416288da","Type":"ContainerDied","Data":"2d4ccfccd1fc1fed31c22081907201ff4e6ae524ec8707652eac01274e109078"}
Oct 31 13:45:20.908775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.908755    4428 scope.go:122] "RemoveContainer" containerID="2d4ccfccd1fc1fed31c22081907201ff4e6ae524ec8707652eac01274e109078"
Oct 31 13:45:20.909202 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:20.909181    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs_odh-ai-gateway-infra(77411480-49b1-49e5-90f0-1fc4416288da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs" podUID="77411480-49b1-49e5-90f0-1fc4416288da"
Oct 31 13:45:20.912802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.912777    4428 generic.go:356] "Generic (PLEG): container finished" podID="680bd885-b3c7-42f5-a3a8-4f2950d0b462" containerID="e5498ae5122afdb74b036f395f0039b1c7203a8a89451ec4476ee0dc8af9610a" exitCode=60
Oct 31 13:45:20.912918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.912847    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp" event={"ID":"680bd885-b3c7-42f5-a3a8-4f2950d0b462","Type":"ContainerDied","Data":"e5498ae5122afdb74b036f395f0039b1c7203a8a89451ec4476ee0dc8af9610a"}
Oct 31 13:45:20.913316 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.913298    4428 scope.go:122] "RemoveContainer" containerID="e5498ae5122afdb74b036f395f0039b1c7203a8a89451ec4476ee0dc8af9610a"
Oct 31 13:45:20.913685 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:20.913655    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp_odh-ai-gateway-infra(680bd885-b3c7-42f5-a3a8-4f2950d0b462)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp" podUID="680bd885-b3c7-42f5-a3a8-4f2950d0b462"
Oct 31 13:45:20.915400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.915379    4428 scope.go:122] "RemoveContainer" containerID="d3908faeff05e0b906b31adfea188260f2250418efad81f9d9a7e2b0e21dba40"
Oct 31 13:45:20.916272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.916238    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7" event={"ID":"e615817e-f1b7-485a-b4d4-b63b54b6db48","Type":"ContainerStarted","Data":"2ed4a2b116ed36ed6f7475d9869d9dbe350a08e306cf8a3db0c864cdba212013"}
Oct 31 13:45:20.952229 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.952090    4428 scope.go:122] "RemoveContainer" containerID="07229caea95318bbdfc954a269a093763ffcfce6dbffa44422cbe87775b61d8a"
Oct 31 13:45:21.000022 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:20.988983    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs"]
Oct 31 13:45:21.034691 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.034663    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8"]
Oct 31 13:45:21.057977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.057946    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp"]
Oct 31 13:45:21.064342 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.064318    4428 scope.go:122] "RemoveContainer" containerID="ff473a4d21cc30d7cd596ca6c96bd79c6d39b0ecaf4690c570084fbb3516235f"
Oct 31 13:45:21.139447 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.138894    4428 scope.go:122] "RemoveContainer" containerID="9fa384df2d11aebb2fba2360adc408bf7f9aa7767046215aceb69aa61e821542"
Oct 31 13:45:21.147752 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.147602    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4"]
Oct 31 13:45:21.218125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.218067    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:21.426979 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.426924    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:21.686248 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.686162    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp"
Oct 31 13:45:21.688359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.688316    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76"
Oct 31 13:45:21.696240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.696176    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw"
Oct 31 13:45:21.731658 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.731625    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67"
Oct 31 13:45:21.846469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.846398    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d507266c-1d74-4d2c-9af8-c9990fe98174-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d507266c-1d74-4d2c-9af8-c9990fe98174-maas-api-service-ca\") pod \"d507266c-1d74-4d2c-9af8-c9990fe98174\" (UID: \"d507266c-1d74-4d2c-9af8-c9990fe98174\") "
Oct 31 13:45:21.846889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.846838    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d507266c-1d74-4d2c-9af8-c9990fe98174-maas-api-service-ca" pod "d507266c-1d74-4d2c-9af8-c9990fe98174" (UID: "d507266c-1d74-4d2c-9af8-c9990fe98174"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:21.847014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.846989    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc233cb4-c11a-4bd5-9abb-951884813235-kube-api-access-4xw2h\" (UniqueName: \"kubernetes.io/projected/dc233cb4-c11a-4bd5-9abb-951884813235-kube-api-access-4xw2h\") pod \"dc233cb4-c11a-4bd5-9abb-951884813235\" (UID: \"dc233cb4-c11a-4bd5-9abb-951884813235\") "
Oct 31 13:45:21.847085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.847043    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc233cb4-c11a-4bd5-9abb-951884813235-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc233cb4-c11a-4bd5-9abb-951884813235-maas-api-service-ca\") pod \"dc233cb4-c11a-4bd5-9abb-951884813235\" (UID: \"dc233cb4-c11a-4bd5-9abb-951884813235\") "
Oct 31 13:45:21.847186 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.847171    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9daa213c-e1ea-473d-8ed6-a2bc15630b58-kube-api-access-m6bdt\" (UniqueName: \"kubernetes.io/projected/9daa213c-e1ea-473d-8ed6-a2bc15630b58-kube-api-access-m6bdt\") pod \"9daa213c-e1ea-473d-8ed6-a2bc15630b58\" (UID: \"9daa213c-e1ea-473d-8ed6-a2bc15630b58\") "
Oct 31 13:45:21.847287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.847262    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d507266c-1d74-4d2c-9af8-c9990fe98174-kube-api-access-2w9ps\" (UniqueName: \"kubernetes.io/projected/d507266c-1d74-4d2c-9af8-c9990fe98174-kube-api-access-2w9ps\") pod \"d507266c-1d74-4d2c-9af8-c9990fe98174\" (UID: \"d507266c-1d74-4d2c-9af8-c9990fe98174\") "
Oct 31 13:45:21.847454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.847436    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9daa213c-e1ea-473d-8ed6-a2bc15630b58-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9daa213c-e1ea-473d-8ed6-a2bc15630b58-maas-api-service-ca\") pod \"9daa213c-e1ea-473d-8ed6-a2bc15630b58\" (UID: \"9daa213c-e1ea-473d-8ed6-a2bc15630b58\") "
Oct 31 13:45:21.847788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.847758    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61fe9f57-eaa6-49c1-9a63-f37485554a23-kube-api-access-zcghq\" (UniqueName: \"kubernetes.io/projected/61fe9f57-eaa6-49c1-9a63-f37485554a23-kube-api-access-zcghq\") pod \"61fe9f57-eaa6-49c1-9a63-f37485554a23\" (UID: \"61fe9f57-eaa6-49c1-9a63-f37485554a23\") "
Oct 31 13:45:21.847990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.847849    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61fe9f57-eaa6-49c1-9a63-f37485554a23-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fe9f57-eaa6-49c1-9a63-f37485554a23-maas-api-service-ca\") pod \"61fe9f57-eaa6-49c1-9a63-f37485554a23\" (UID: \"61fe9f57-eaa6-49c1-9a63-f37485554a23\") "
Oct 31 13:45:21.849242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.847438    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc233cb4-c11a-4bd5-9abb-951884813235-maas-api-service-ca" pod "dc233cb4-c11a-4bd5-9abb-951884813235" (UID: "dc233cb4-c11a-4bd5-9abb-951884813235"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:21.849242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.848315    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9daa213c-e1ea-473d-8ed6-a2bc15630b58-maas-api-service-ca" pod "9daa213c-e1ea-473d-8ed6-a2bc15630b58" (UID: "9daa213c-e1ea-473d-8ed6-a2bc15630b58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:21.849242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.848718    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61fe9f57-eaa6-49c1-9a63-f37485554a23-maas-api-service-ca" pod "61fe9f57-eaa6-49c1-9a63-f37485554a23" (UID: "61fe9f57-eaa6-49c1-9a63-f37485554a23"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:21.849242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.849158    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc233cb4-c11a-4bd5-9abb-951884813235-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:21.849242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.849181    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9daa213c-e1ea-473d-8ed6-a2bc15630b58-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:21.849242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.849196    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fe9f57-eaa6-49c1-9a63-f37485554a23-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:21.849242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.849211    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d507266c-1d74-4d2c-9af8-c9990fe98174-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:21.850147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.850119    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc233cb4-c11a-4bd5-9abb-951884813235-kube-api-access-4xw2h" pod "dc233cb4-c11a-4bd5-9abb-951884813235" (UID: "dc233cb4-c11a-4bd5-9abb-951884813235"). InnerVolumeSpecName "kube-api-access-4xw2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:21.850231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.850190    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d507266c-1d74-4d2c-9af8-c9990fe98174-kube-api-access-2w9ps" pod "d507266c-1d74-4d2c-9af8-c9990fe98174" (UID: "d507266c-1d74-4d2c-9af8-c9990fe98174"). InnerVolumeSpecName "kube-api-access-2w9ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:21.851967 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.851940    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9daa213c-e1ea-473d-8ed6-a2bc15630b58-kube-api-access-m6bdt" pod "9daa213c-e1ea-473d-8ed6-a2bc15630b58" (UID: "9daa213c-e1ea-473d-8ed6-a2bc15630b58"). InnerVolumeSpecName "kube-api-access-m6bdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:21.852068 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.851992    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61fe9f57-eaa6-49c1-9a63-f37485554a23-kube-api-access-zcghq" pod "61fe9f57-eaa6-49c1-9a63-f37485554a23" (UID: "61fe9f57-eaa6-49c1-9a63-f37485554a23"). InnerVolumeSpecName "kube-api-access-zcghq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:21.942547 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.942439    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67"
Oct 31 13:45:21.946917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.942448    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67" event={"ID":"61fe9f57-eaa6-49c1-9a63-f37485554a23","Type":"ContainerDied","Data":"79fa73e23286bc145712d6d4cde5339cbc17bd2bdcabdd83b13808dd417dc315"}
Oct 31 13:45:21.946917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.943381    4428 scope.go:122] "RemoveContainer" containerID="38e4374d4fc7e0bf00aadb4f7ee5d5a2281a49d45b18e9e3bf0706f13f089bbb"
Oct 31 13:45:21.952163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.952128    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcghq\" (UniqueName: \"kubernetes.io/projected/61fe9f57-eaa6-49c1-9a63-f37485554a23-kube-api-access-zcghq\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:21.952163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.952165    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xw2h\" (UniqueName: \"kubernetes.io/projected/dc233cb4-c11a-4bd5-9abb-951884813235-kube-api-access-4xw2h\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:21.952391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.952179    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6bdt\" (UniqueName: \"kubernetes.io/projected/9daa213c-e1ea-473d-8ed6-a2bc15630b58-kube-api-access-m6bdt\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:21.952391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.952194    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w9ps\" (UniqueName: \"kubernetes.io/projected/d507266c-1d74-4d2c-9af8-c9990fe98174-kube-api-access-2w9ps\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:21.959942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.959920    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp"
Oct 31 13:45:21.960062 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.959911    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp" event={"ID":"9daa213c-e1ea-473d-8ed6-a2bc15630b58","Type":"ContainerDied","Data":"c03166d87cc055a621b8e13fb439de4b3633931a90e6eeddac69a8a694747ad7"}
Oct 31 13:45:21.960131 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.960116    4428 scope.go:122] "RemoveContainer" containerID="a952a21130269853de2d0afe5cf9c89c11263fe17b0ee8bfd153ecd8bedcfdf6"
Oct 31 13:45:21.962238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.962216    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw"
Oct 31 13:45:21.962424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.962285    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw" event={"ID":"dc233cb4-c11a-4bd5-9abb-951884813235","Type":"ContainerDied","Data":"fc6ec93576cca0cf75b916fe4e15e7c3e1320ed9f04ebb244e41aaae753a4684"}
Oct 31 13:45:21.967079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.967052    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76" event={"ID":"d507266c-1d74-4d2c-9af8-c9990fe98174","Type":"ContainerDied","Data":"683b298b886da9b556fce5954f65a1873047e87a5e1ba560856c29b0831a47f0"}
Oct 31 13:45:21.967203 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.967158    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76"
Oct 31 13:45:21.986187 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:21.986170    4428 scope.go:122] "RemoveContainer" containerID="787bb9c84d3091c6300fe9ebfda8f3f2ebd74a5b7897b527a879a7a36b693b43"
Oct 31 13:45:22.006742 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.006601    4428 generic.go:356] "Generic (PLEG): container finished" podID="e615817e-f1b7-485a-b4d4-b63b54b6db48" containerID="2ed4a2b116ed36ed6f7475d9869d9dbe350a08e306cf8a3db0c864cdba212013" exitCode=60
Oct 31 13:45:22.009908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.006990    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7" event={"ID":"e615817e-f1b7-485a-b4d4-b63b54b6db48","Type":"ContainerDied","Data":"2ed4a2b116ed36ed6f7475d9869d9dbe350a08e306cf8a3db0c864cdba212013"}
Oct 31 13:45:22.009908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.007810    4428 scope.go:122] "RemoveContainer" containerID="2ed4a2b116ed36ed6f7475d9869d9dbe350a08e306cf8a3db0c864cdba212013"
Oct 31 13:45:22.009908 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:22.008809    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7_odh-ai-gateway-infra(e615817e-f1b7-485a-b4d4-b63b54b6db48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7" podUID="e615817e-f1b7-485a-b4d4-b63b54b6db48"
Oct 31 13:45:22.042647 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.038966    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8"]
Oct 31 13:45:22.070855 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.065202    4428 scope.go:122] "RemoveContainer" containerID="24ad68245cb5e1dc57e40bfbbab55fc9fdb775a855f762b357953b4e854df412"
Oct 31 13:45:22.104190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.104158    4428 scope.go:122] "RemoveContainer" containerID="72afb27e590373ac7f90f3bc05927562a4b4661a9b07a93920bd9997fe37c26d"
Oct 31 13:45:22.127007 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.126969    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67"]
Oct 31 13:45:22.128117 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.128096    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aabeef-29890905-xcn67"]
Oct 31 13:45:22.156050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.156009    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7"]
Oct 31 13:45:22.218808 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.218766    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw"]
Oct 31 13:45:22.230491 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.230460    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0d835b-29890905-lcxzw"]
Oct 31 13:45:22.234539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.234273    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4"
Oct 31 13:45:22.267345 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.267310    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76"]
Oct 31 13:45:22.275296 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.275263    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cae9ec-29890905-lnd76"]
Oct 31 13:45:22.324825 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.324793    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp"]
Oct 31 13:45:22.330200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.330138    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-61f295-29890905-9zgxp"]
Oct 31 13:45:22.393611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.393581    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9485106e-58f9-4b42-ae34-0136f6345af2-kube-api-access-bmvrc\" (UniqueName: \"kubernetes.io/projected/9485106e-58f9-4b42-ae34-0136f6345af2-kube-api-access-bmvrc\") pod \"9485106e-58f9-4b42-ae34-0136f6345af2\" (UID: \"9485106e-58f9-4b42-ae34-0136f6345af2\") "
Oct 31 13:45:22.394731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.394699    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9485106e-58f9-4b42-ae34-0136f6345af2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9485106e-58f9-4b42-ae34-0136f6345af2-maas-api-service-ca\") pod \"9485106e-58f9-4b42-ae34-0136f6345af2\" (UID: \"9485106e-58f9-4b42-ae34-0136f6345af2\") "
Oct 31 13:45:22.396353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.396321    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9485106e-58f9-4b42-ae34-0136f6345af2-maas-api-service-ca" pod "9485106e-58f9-4b42-ae34-0136f6345af2" (UID: "9485106e-58f9-4b42-ae34-0136f6345af2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:22.407285 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.407081    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9485106e-58f9-4b42-ae34-0136f6345af2-kube-api-access-bmvrc" pod "9485106e-58f9-4b42-ae34-0136f6345af2" (UID: "9485106e-58f9-4b42-ae34-0136f6345af2"). InnerVolumeSpecName "kube-api-access-bmvrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:22.417758 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.417734    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp"
Oct 31 13:45:22.420664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.420640    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8"
Oct 31 13:45:22.426076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.426051    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs"
Oct 31 13:45:22.432026 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.431994    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8"
Oct 31 13:45:22.497570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.497461    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmvrc\" (UniqueName: \"kubernetes.io/projected/9485106e-58f9-4b42-ae34-0136f6345af2-kube-api-access-bmvrc\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:22.497570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.497495    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9485106e-58f9-4b42-ae34-0136f6345af2-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:22.575319 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.575276    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"]
Oct 31 13:45:22.575587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.575559    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv" podUID="e899d7ff-9dd2-48d8-b537-f03e45b6d1c2" containerName="discovery" containerID="cri-o://ad97f22244d8b644cdb74aa025e760f718f0625af825cbee5003f97d808b75aa" gracePeriod=30
Oct 31 13:45:22.598902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.598414    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77411480-49b1-49e5-90f0-1fc4416288da-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77411480-49b1-49e5-90f0-1fc4416288da-maas-api-service-ca\") pod \"77411480-49b1-49e5-90f0-1fc4416288da\" (UID: \"77411480-49b1-49e5-90f0-1fc4416288da\") "
Oct 31 13:45:22.598902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.598488    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb5549e-6008-46a8-8e55-3b37cb62cd60-kube-api-access-8sfnj\" (UniqueName: \"kubernetes.io/projected/6bb5549e-6008-46a8-8e55-3b37cb62cd60-kube-api-access-8sfnj\") pod \"6bb5549e-6008-46a8-8e55-3b37cb62cd60\" (UID: \"6bb5549e-6008-46a8-8e55-3b37cb62cd60\") "
Oct 31 13:45:22.598902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.598562    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d9f9bc-22ec-4943-8a8c-9ed7abe7af98-kube-api-access-85ssh\" (UniqueName: \"kubernetes.io/projected/55d9f9bc-22ec-4943-8a8c-9ed7abe7af98-kube-api-access-85ssh\") pod \"55d9f9bc-22ec-4943-8a8c-9ed7abe7af98\" (UID: \"55d9f9bc-22ec-4943-8a8c-9ed7abe7af98\") "
Oct 31 13:45:22.599241 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.599214    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77411480-49b1-49e5-90f0-1fc4416288da-maas-api-service-ca" pod "77411480-49b1-49e5-90f0-1fc4416288da" (UID: "77411480-49b1-49e5-90f0-1fc4416288da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:22.599376 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.599271    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77411480-49b1-49e5-90f0-1fc4416288da-kube-api-access-6498d\" (UniqueName: \"kubernetes.io/projected/77411480-49b1-49e5-90f0-1fc4416288da-kube-api-access-6498d\") pod \"77411480-49b1-49e5-90f0-1fc4416288da\" (UID: \"77411480-49b1-49e5-90f0-1fc4416288da\") "
Oct 31 13:45:22.599376 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.599357    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55d9f9bc-22ec-4943-8a8c-9ed7abe7af98-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d9f9bc-22ec-4943-8a8c-9ed7abe7af98-maas-api-service-ca\") pod \"55d9f9bc-22ec-4943-8a8c-9ed7abe7af98\" (UID: \"55d9f9bc-22ec-4943-8a8c-9ed7abe7af98\") "
Oct 31 13:45:22.599500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.599394    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb5549e-6008-46a8-8e55-3b37cb62cd60-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb5549e-6008-46a8-8e55-3b37cb62cd60-maas-api-service-ca\") pod \"6bb5549e-6008-46a8-8e55-3b37cb62cd60\" (UID: \"6bb5549e-6008-46a8-8e55-3b37cb62cd60\") "
Oct 31 13:45:22.599500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.599473    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680bd885-b3c7-42f5-a3a8-4f2950d0b462-kube-api-access-q8tkm\" (UniqueName: \"kubernetes.io/projected/680bd885-b3c7-42f5-a3a8-4f2950d0b462-kube-api-access-q8tkm\") pod \"680bd885-b3c7-42f5-a3a8-4f2950d0b462\" (UID: \"680bd885-b3c7-42f5-a3a8-4f2950d0b462\") "
Oct 31 13:45:22.599621 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.599566    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/680bd885-b3c7-42f5-a3a8-4f2950d0b462-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/680bd885-b3c7-42f5-a3a8-4f2950d0b462-maas-api-service-ca\") pod \"680bd885-b3c7-42f5-a3a8-4f2950d0b462\" (UID: \"680bd885-b3c7-42f5-a3a8-4f2950d0b462\") "
Oct 31 13:45:22.600418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.600394    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77411480-49b1-49e5-90f0-1fc4416288da-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:22.600791 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.600766    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680bd885-b3c7-42f5-a3a8-4f2950d0b462-maas-api-service-ca" pod "680bd885-b3c7-42f5-a3a8-4f2950d0b462" (UID: "680bd885-b3c7-42f5-a3a8-4f2950d0b462"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:22.604524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.601580    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb5549e-6008-46a8-8e55-3b37cb62cd60-kube-api-access-8sfnj" pod "6bb5549e-6008-46a8-8e55-3b37cb62cd60" (UID: "6bb5549e-6008-46a8-8e55-3b37cb62cd60"). InnerVolumeSpecName "kube-api-access-8sfnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:22.604524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.602267    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb5549e-6008-46a8-8e55-3b37cb62cd60-maas-api-service-ca" pod "6bb5549e-6008-46a8-8e55-3b37cb62cd60" (UID: "6bb5549e-6008-46a8-8e55-3b37cb62cd60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:22.605043 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.605018    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77411480-49b1-49e5-90f0-1fc4416288da-kube-api-access-6498d" pod "77411480-49b1-49e5-90f0-1fc4416288da" (UID: "77411480-49b1-49e5-90f0-1fc4416288da"). InnerVolumeSpecName "kube-api-access-6498d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:22.605646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.605137    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680bd885-b3c7-42f5-a3a8-4f2950d0b462-kube-api-access-q8tkm" pod "680bd885-b3c7-42f5-a3a8-4f2950d0b462" (UID: "680bd885-b3c7-42f5-a3a8-4f2950d0b462"). InnerVolumeSpecName "kube-api-access-q8tkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:22.609732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.609705    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d9f9bc-22ec-4943-8a8c-9ed7abe7af98-kube-api-access-85ssh" pod "55d9f9bc-22ec-4943-8a8c-9ed7abe7af98" (UID: "55d9f9bc-22ec-4943-8a8c-9ed7abe7af98"). InnerVolumeSpecName "kube-api-access-85ssh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:22.614311 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.614281    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d9f9bc-22ec-4943-8a8c-9ed7abe7af98-maas-api-service-ca" pod "55d9f9bc-22ec-4943-8a8c-9ed7abe7af98" (UID: "55d9f9bc-22ec-4943-8a8c-9ed7abe7af98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:22.662596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.662561    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"]
Oct 31 13:45:22.663466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663445    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9daa213c-e1ea-473d-8ed6-a2bc15630b58" containerName="cleanup"
Oct 31 13:45:22.663466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663464    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9daa213c-e1ea-473d-8ed6-a2bc15630b58" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663476    4428 cpu_manager.go:414] "containerMap: removing container" podUID="6bb5549e-6008-46a8-8e55-3b37cb62cd60" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663481    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb5549e-6008-46a8-8e55-3b37cb62cd60" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663492    4428 cpu_manager.go:414] "containerMap: removing container" podUID="680bd885-b3c7-42f5-a3a8-4f2950d0b462" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663498    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680bd885-b3c7-42f5-a3a8-4f2950d0b462" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663506    4428 cpu_manager.go:414] "containerMap: removing container" podUID="77411480-49b1-49e5-90f0-1fc4416288da" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663510    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77411480-49b1-49e5-90f0-1fc4416288da" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663518    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d507266c-1d74-4d2c-9af8-c9990fe98174" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663522    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d507266c-1d74-4d2c-9af8-c9990fe98174" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663529    4428 cpu_manager.go:414] "containerMap: removing container" podUID="61fe9f57-eaa6-49c1-9a63-f37485554a23" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663533    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fe9f57-eaa6-49c1-9a63-f37485554a23" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663540    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d507266c-1d74-4d2c-9af8-c9990fe98174" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663545    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d507266c-1d74-4d2c-9af8-c9990fe98174" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663556    4428 cpu_manager.go:414] "containerMap: removing container" podUID="dc233cb4-c11a-4bd5-9abb-951884813235" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663560    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc233cb4-c11a-4bd5-9abb-951884813235" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663567    4428 cpu_manager.go:414] "containerMap: removing container" podUID="77411480-49b1-49e5-90f0-1fc4416288da" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663572    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77411480-49b1-49e5-90f0-1fc4416288da" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663582    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9485106e-58f9-4b42-ae34-0136f6345af2" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663589    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9485106e-58f9-4b42-ae34-0136f6345af2" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663608    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9daa213c-e1ea-473d-8ed6-a2bc15630b58" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663612    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9daa213c-e1ea-473d-8ed6-a2bc15630b58" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663621    4428 cpu_manager.go:414] "containerMap: removing container" podUID="680bd885-b3c7-42f5-a3a8-4f2950d0b462" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663625    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680bd885-b3c7-42f5-a3a8-4f2950d0b462" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663633    4428 cpu_manager.go:414] "containerMap: removing container" podUID="dc233cb4-c11a-4bd5-9abb-951884813235" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663637    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc233cb4-c11a-4bd5-9abb-951884813235" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663651    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9daa213c-e1ea-473d-8ed6-a2bc15630b58" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663655    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9daa213c-e1ea-473d-8ed6-a2bc15630b58" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663662    4428 cpu_manager.go:414] "containerMap: removing container" podUID="55d9f9bc-22ec-4943-8a8c-9ed7abe7af98" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663666    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d9f9bc-22ec-4943-8a8c-9ed7abe7af98" containerName="cleanup"
Oct 31 13:45:22.663654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663675    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d507266c-1d74-4d2c-9af8-c9990fe98174" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663679    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d507266c-1d74-4d2c-9af8-c9990fe98174" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663690    4428 cpu_manager.go:414] "containerMap: removing container" podUID="61fe9f57-eaa6-49c1-9a63-f37485554a23" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663694    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fe9f57-eaa6-49c1-9a63-f37485554a23" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663701    4428 cpu_manager.go:414] "containerMap: removing container" podUID="77411480-49b1-49e5-90f0-1fc4416288da" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663705    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77411480-49b1-49e5-90f0-1fc4416288da" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663713    4428 cpu_manager.go:414] "containerMap: removing container" podUID="55d9f9bc-22ec-4943-8a8c-9ed7abe7af98" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663717    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d9f9bc-22ec-4943-8a8c-9ed7abe7af98" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663723    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9485106e-58f9-4b42-ae34-0136f6345af2" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663727    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9485106e-58f9-4b42-ae34-0136f6345af2" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663733    4428 cpu_manager.go:414] "containerMap: removing container" podUID="55d9f9bc-22ec-4943-8a8c-9ed7abe7af98" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663737    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d9f9bc-22ec-4943-8a8c-9ed7abe7af98" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663745    4428 cpu_manager.go:414] "containerMap: removing container" podUID="6bb5549e-6008-46a8-8e55-3b37cb62cd60" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663749    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb5549e-6008-46a8-8e55-3b37cb62cd60" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663880    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb5549e-6008-46a8-8e55-3b37cb62cd60" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663892    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fe9f57-eaa6-49c1-9a63-f37485554a23" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663900    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9daa213c-e1ea-473d-8ed6-a2bc15630b58" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663907    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d9f9bc-22ec-4943-8a8c-9ed7abe7af98" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663916    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d507266c-1d74-4d2c-9af8-c9990fe98174" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663923    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc233cb4-c11a-4bd5-9abb-951884813235" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663929    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9485106e-58f9-4b42-ae34-0136f6345af2" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663936    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fe9f57-eaa6-49c1-9a63-f37485554a23" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663941    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="77411480-49b1-49e5-90f0-1fc4416288da" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663947    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="680bd885-b3c7-42f5-a3a8-4f2950d0b462" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663951    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d9f9bc-22ec-4943-8a8c-9ed7abe7af98" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663957    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9485106e-58f9-4b42-ae34-0136f6345af2" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663962    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d9f9bc-22ec-4943-8a8c-9ed7abe7af98" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663969    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9daa213c-e1ea-473d-8ed6-a2bc15630b58" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663976    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb5549e-6008-46a8-8e55-3b37cb62cd60" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663982    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="680bd885-b3c7-42f5-a3a8-4f2950d0b462" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663988    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d507266c-1d74-4d2c-9af8-c9990fe98174" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.663993    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fe9f57-eaa6-49c1-9a63-f37485554a23" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.664002    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d507266c-1d74-4d2c-9af8-c9990fe98174" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.664010    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="77411480-49b1-49e5-90f0-1fc4416288da" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.664028    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="680bd885-b3c7-42f5-a3a8-4f2950d0b462" containerName="cleanup"
Oct 31 13:45:22.665386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.664036    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc233cb4-c11a-4bd5-9abb-951884813235" containerName="cleanup"
Oct 31 13:45:22.668565 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.668545    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"
Oct 31 13:45:22.674448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.674107    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:45:22.703460 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.703429    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6498d\" (UniqueName: \"kubernetes.io/projected/77411480-49b1-49e5-90f0-1fc4416288da-kube-api-access-6498d\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:22.703720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.703700    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d9f9bc-22ec-4943-8a8c-9ed7abe7af98-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:22.703846 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.703833    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb5549e-6008-46a8-8e55-3b37cb62cd60-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:22.703977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.703964    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8tkm\" (UniqueName: \"kubernetes.io/projected/680bd885-b3c7-42f5-a3a8-4f2950d0b462-kube-api-access-q8tkm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:22.704098 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.704082    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/680bd885-b3c7-42f5-a3a8-4f2950d0b462-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:22.704206 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.704193    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sfnj\" (UniqueName: \"kubernetes.io/projected/6bb5549e-6008-46a8-8e55-3b37cb62cd60-kube-api-access-8sfnj\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:22.704313 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.704301    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85ssh\" (UniqueName: \"kubernetes.io/projected/55d9f9bc-22ec-4943-8a8c-9ed7abe7af98-kube-api-access-85ssh\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:22.716940 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.716840    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"]
Oct 31 13:45:22.770243 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.770153    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61fe9f57-eaa6-49c1-9a63-f37485554a23" path="/var/lib/kubelet/pods/61fe9f57-eaa6-49c1-9a63-f37485554a23/volumes"
Oct 31 13:45:22.770765 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.770737    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9daa213c-e1ea-473d-8ed6-a2bc15630b58" path="/var/lib/kubelet/pods/9daa213c-e1ea-473d-8ed6-a2bc15630b58/volumes"
Oct 31 13:45:22.771329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.771307    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d507266c-1d74-4d2c-9af8-c9990fe98174" path="/var/lib/kubelet/pods/d507266c-1d74-4d2c-9af8-c9990fe98174/volumes"
Oct 31 13:45:22.772357 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.772335    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc233cb4-c11a-4bd5-9abb-951884813235" path="/var/lib/kubelet/pods/dc233cb4-c11a-4bd5-9abb-951884813235/volumes"
Oct 31 13:45:22.806192 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.806131    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg56z\" (UniqueName: \"kubernetes.io/projected/3e053af8-843e-4184-8a8d-770834fc5cf2-kube-api-access-cg56z\") pod \"maas-api-revoke-sub-9945cb57-7dtx4\" (UID: \"3e053af8-843e-4184-8a8d-770834fc5cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"
Oct 31 13:45:22.806602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.806577    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e053af8-843e-4184-8a8d-770834fc5cf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9945cb57-7dtx4\" (UID: \"3e053af8-843e-4184-8a8d-770834fc5cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"
Oct 31 13:45:22.806888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.806848    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e053af8-843e-4184-8a8d-770834fc5cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9945cb57-7dtx4\" (UID: \"3e053af8-843e-4184-8a8d-770834fc5cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"
Oct 31 13:45:22.908453 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.908374    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e053af8-843e-4184-8a8d-770834fc5cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9945cb57-7dtx4\" (UID: \"3e053af8-843e-4184-8a8d-770834fc5cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"
Oct 31 13:45:22.908648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.908594    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg56z\" (UniqueName: \"kubernetes.io/projected/3e053af8-843e-4184-8a8d-770834fc5cf2-kube-api-access-cg56z\") pod \"maas-api-revoke-sub-9945cb57-7dtx4\" (UID: \"3e053af8-843e-4184-8a8d-770834fc5cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"
Oct 31 13:45:22.908839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.908821    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e053af8-843e-4184-8a8d-770834fc5cf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9945cb57-7dtx4\" (UID: \"3e053af8-843e-4184-8a8d-770834fc5cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"
Oct 31 13:45:22.909258 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.909188    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e053af8-843e-4184-8a8d-770834fc5cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9945cb57-7dtx4\" (UID: \"3e053af8-843e-4184-8a8d-770834fc5cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"
Oct 31 13:45:22.911861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.911837    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e053af8-843e-4184-8a8d-770834fc5cf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9945cb57-7dtx4\" (UID: \"3e053af8-843e-4184-8a8d-770834fc5cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"
Oct 31 13:45:22.919651 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.919628    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:45:22.921734 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:22.921711    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg56z\" (UniqueName: \"kubernetes.io/projected/3e053af8-843e-4184-8a8d-770834fc5cf2-kube-api-access-cg56z\") pod \"maas-api-revoke-sub-9945cb57-7dtx4\" (UID: \"3e053af8-843e-4184-8a8d-770834fc5cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"
Oct 31 13:45:23.010115 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.010080    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-csr-ca-configmap\") pod \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") "
Oct 31 13:45:23.010562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.010184    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-token\" (UniqueName: \"kubernetes.io/projected/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-token\") pod \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") "
Oct 31 13:45:23.010562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.010270    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-csr-dns-cert\") pod \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") "
Oct 31 13:45:23.010562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.010319    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-kube-api-access-f8s2s\" (UniqueName: \"kubernetes.io/projected/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-kube-api-access-f8s2s\") pod \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") "
Oct 31 13:45:23.010562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.010415    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-kubeconfig\") pod \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") "
Oct 31 13:45:23.010745 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.010558    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-csr-ca-configmap" pod "e899d7ff-9dd2-48d8-b537-f03e45b6d1c2" (UID: "e899d7ff-9dd2-48d8-b537-f03e45b6d1c2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:23.010745 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.010624    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-cacerts\" (UniqueName: \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-cacerts\") pod \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") "
Oct 31 13:45:23.010852 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.010771    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-local-certs\") pod \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\" (UID: \"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2\") "
Oct 31 13:45:23.011691 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.011615    4428 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-csr-ca-configmap\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:23.013602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.013572    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-kubeconfig" pod "e899d7ff-9dd2-48d8-b537-f03e45b6d1c2" (UID: "e899d7ff-9dd2-48d8-b537-f03e45b6d1c2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:45:23.013717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.013632    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-cacerts" pod "e899d7ff-9dd2-48d8-b537-f03e45b6d1c2" (UID: "e899d7ff-9dd2-48d8-b537-f03e45b6d1c2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:45:23.013717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.013693    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-token" pod "e899d7ff-9dd2-48d8-b537-f03e45b6d1c2" (UID: "e899d7ff-9dd2-48d8-b537-f03e45b6d1c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:23.014134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.014108    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-kube-api-access-f8s2s" pod "e899d7ff-9dd2-48d8-b537-f03e45b6d1c2" (UID: "e899d7ff-9dd2-48d8-b537-f03e45b6d1c2"). InnerVolumeSpecName "kube-api-access-f8s2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:23.014432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.014415    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-csr-dns-cert" pod "e899d7ff-9dd2-48d8-b537-f03e45b6d1c2" (UID: "e899d7ff-9dd2-48d8-b537-f03e45b6d1c2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:45:23.014829 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.014802    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-local-certs" pod "e899d7ff-9dd2-48d8-b537-f03e45b6d1c2" (UID: "e899d7ff-9dd2-48d8-b537-f03e45b6d1c2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:23.015996 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.015969    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"
Oct 31 13:45:23.021095 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.021034    4428 generic.go:356] "Generic (PLEG): container finished" podID="e899d7ff-9dd2-48d8-b537-f03e45b6d1c2" containerID="ad97f22244d8b644cdb74aa025e760f718f0625af825cbee5003f97d808b75aa" exitCode=0
Oct 31 13:45:23.021235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.021129    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv" event={"ID":"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2","Type":"ContainerDied","Data":"ad97f22244d8b644cdb74aa025e760f718f0625af825cbee5003f97d808b75aa"}
Oct 31 13:45:23.021235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.021181    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv" event={"ID":"e899d7ff-9dd2-48d8-b537-f03e45b6d1c2","Type":"ContainerDied","Data":"96853bdb9097a411ea49ff7ad8a61122437b338c8fb3ef266b3de714a3f2d64a"}
Oct 31 13:45:23.021235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.021208    4428 scope.go:122] "RemoveContainer" containerID="ad97f22244d8b644cdb74aa025e760f718f0625af825cbee5003f97d808b75aa"
Oct 31 13:45:23.021527 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.021259    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"
Oct 31 13:45:23.045572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.045537    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4" event={"ID":"9485106e-58f9-4b42-ae34-0136f6345af2","Type":"ContainerDied","Data":"f4ff7173cf463fa77294cb5afd3e25bd0fbc5e211a501b20b208a215eb151f31"}
Oct 31 13:45:23.045711 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.045652    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4"
Oct 31 13:45:23.051115 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.051085    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8" event={"ID":"55d9f9bc-22ec-4943-8a8c-9ed7abe7af98","Type":"ContainerDied","Data":"216f3d035980cb6319b9302695212b44327305e649f52dccbe1df92f5133fe03"}
Oct 31 13:45:23.051282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.051198    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8"
Oct 31 13:45:23.077360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.077108    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8" event={"ID":"6bb5549e-6008-46a8-8e55-3b37cb62cd60","Type":"ContainerDied","Data":"9ecf1e5a59950ed2fdb00e1fcc26a6b88c97862dc8c91221bb6f9db018502ea8"}
Oct 31 13:45:23.077360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.077253    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8"
Oct 31 13:45:23.083838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.083800    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs" event={"ID":"77411480-49b1-49e5-90f0-1fc4416288da","Type":"ContainerDied","Data":"4a68214c6cfeb32c88fe5e6c59ee3be86094019df5ca2fc8f4fd6f81b79c0dae"}
Oct 31 13:45:23.084089 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.083942    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs"
Oct 31 13:45:23.087575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.087548    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp" event={"ID":"680bd885-b3c7-42f5-a3a8-4f2950d0b462","Type":"ContainerDied","Data":"7c0faf35751b30c038fb4f3c911cceadf8fbd4003e54bb678e78fb486cc03f20"}
Oct 31 13:45:23.087708 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.087627    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp"
Oct 31 13:45:23.111234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.111216    4428 scope.go:122] "RemoveContainer" containerID="ad97f22244d8b644cdb74aa025e760f718f0625af825cbee5003f97d808b75aa"
Oct 31 13:45:23.116198 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.116169    4428 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-cacerts\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:23.116417 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.116399    4428 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-local-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:23.116530 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.116515    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:23.116645 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.116630    4428 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-csr-dns-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:23.118112 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.118092    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8s2s\" (UniqueName: \"kubernetes.io/projected/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-kube-api-access-f8s2s\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:23.118285 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.118266    4428 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2-istio-kubeconfig\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:23.124789 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:23.123596    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad97f22244d8b644cdb74aa025e760f718f0625af825cbee5003f97d808b75aa\": container with ID starting with ad97f22244d8b644cdb74aa025e760f718f0625af825cbee5003f97d808b75aa not found: ID does not exist" containerID="ad97f22244d8b644cdb74aa025e760f718f0625af825cbee5003f97d808b75aa"
Oct 31 13:45:23.124789 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.123643    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad97f22244d8b644cdb74aa025e760f718f0625af825cbee5003f97d808b75aa"} err="failed to get container status \"ad97f22244d8b644cdb74aa025e760f718f0625af825cbee5003f97d808b75aa\": rpc error: code = NotFound desc = could not find container \"ad97f22244d8b644cdb74aa025e760f718f0625af825cbee5003f97d808b75aa\": container with ID starting with ad97f22244d8b644cdb74aa025e760f718f0625af825cbee5003f97d808b75aa not found: ID does not exist"
Oct 31 13:45:23.124789 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.123676    4428 scope.go:122] "RemoveContainer" containerID="49f403cfdba672172e5c15f6d6b85daa8f5c2983187892f34e50034c8cf44df6"
Oct 31 13:45:23.158534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.158505    4428 scope.go:122] "RemoveContainer" containerID="51ee443bec390fd6bba8e12aa3e9e65b6e6a8bb08994333dede7d58ed9aaca86"
Oct 31 13:45:23.166164 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.166136    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8"]
Oct 31 13:45:23.173122 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.173095    4428 scope.go:122] "RemoveContainer" containerID="27a4efe978c1db6ed5aed81873b84e475064e1f86a1b31c68b53b678f14ebf14"
Oct 31 13:45:23.176583 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.176551    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4f340c-29890905-s72d8"]
Oct 31 13:45:23.184957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.184938    4428 scope.go:122] "RemoveContainer" containerID="2d4ccfccd1fc1fed31c22081907201ff4e6ae524ec8707652eac01274e109078"
Oct 31 13:45:23.194339 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.194314    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp"]
Oct 31 13:45:23.202615 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.202455    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890905-xmfmp"]
Oct 31 13:45:23.213358 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.213336    4428 scope.go:122] "RemoveContainer" containerID="e5498ae5122afdb74b036f395f0039b1c7203a8a89451ec4476ee0dc8af9610a"
Oct 31 13:45:23.219969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.219937    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8"]
Oct 31 13:45:23.271811 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.271738    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:23.293817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.293782    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890905-s6nd8"]
Oct 31 13:45:23.331361 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.331309    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"]
Oct 31 13:45:23.333991 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.333964    4428 cpu_manager.go:414] "containerMap: removing container" podUID="6bb5549e-6008-46a8-8e55-3b37cb62cd60" containerName="cleanup"
Oct 31 13:45:23.351209 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.351171    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb5549e-6008-46a8-8e55-3b37cb62cd60" containerName="cleanup"
Oct 31 13:45:23.351657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.351633    4428 cpu_manager.go:414] "containerMap: removing container" podUID="61fe9f57-eaa6-49c1-9a63-f37485554a23" containerName="cleanup"
Oct 31 13:45:23.351813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.351800    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fe9f57-eaa6-49c1-9a63-f37485554a23" containerName="cleanup"
Oct 31 13:45:23.352239 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.352219    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e899d7ff-9dd2-48d8-b537-f03e45b6d1c2" containerName="discovery"
Oct 31 13:45:23.353216 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.353199    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e899d7ff-9dd2-48d8-b537-f03e45b6d1c2" containerName="discovery"
Oct 31 13:45:23.353368 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.353355    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9485106e-58f9-4b42-ae34-0136f6345af2" containerName="cleanup"
Oct 31 13:45:23.353470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.353459    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9485106e-58f9-4b42-ae34-0136f6345af2" containerName="cleanup"
Oct 31 13:45:23.353593 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.353582    4428 cpu_manager.go:414] "containerMap: removing container" podUID="680bd885-b3c7-42f5-a3a8-4f2950d0b462" containerName="cleanup"
Oct 31 13:45:23.353681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.353669    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680bd885-b3c7-42f5-a3a8-4f2950d0b462" containerName="cleanup"
Oct 31 13:45:23.353775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.353765    4428 cpu_manager.go:414] "containerMap: removing container" podUID="dc233cb4-c11a-4bd5-9abb-951884813235" containerName="cleanup"
Oct 31 13:45:23.353853 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.353843    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc233cb4-c11a-4bd5-9abb-951884813235" containerName="cleanup"
Oct 31 13:45:23.354226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.354208    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9485106e-58f9-4b42-ae34-0136f6345af2" containerName="cleanup"
Oct 31 13:45:23.354341 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.354330    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb5549e-6008-46a8-8e55-3b37cb62cd60" containerName="cleanup"
Oct 31 13:45:23.354446 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.354435    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e899d7ff-9dd2-48d8-b537-f03e45b6d1c2" containerName="discovery"
Oct 31 13:45:23.354532 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.354522    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="77411480-49b1-49e5-90f0-1fc4416288da" containerName="cleanup"
Oct 31 13:45:23.354635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.354624    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9daa213c-e1ea-473d-8ed6-a2bc15630b58" containerName="cleanup"
Oct 31 13:45:23.354723 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.354713    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc233cb4-c11a-4bd5-9abb-951884813235" containerName="cleanup"
Oct 31 13:45:23.363159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.363134    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4"]
Oct 31 13:45:23.363481 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.363467    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890905-blwv4"]
Oct 31 13:45:23.364692 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.363426    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"
Oct 31 13:45:23.377595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.377545    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"]
Oct 31 13:45:23.387677 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.385949    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs"]
Oct 31 13:45:23.393660 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.393079    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890905-7q6fs"]
Oct 31 13:45:23.398098 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.398072    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"]
Oct 31 13:45:23.402533 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.402509    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wb6gv"]
Oct 31 13:45:23.426492 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.426460    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7"
Oct 31 13:45:23.531627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.531546    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e615817e-f1b7-485a-b4d4-b63b54b6db48-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e615817e-f1b7-485a-b4d4-b63b54b6db48-maas-api-service-ca\") pod \"e615817e-f1b7-485a-b4d4-b63b54b6db48\" (UID: \"e615817e-f1b7-485a-b4d4-b63b54b6db48\") "
Oct 31 13:45:23.531627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.531590    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e615817e-f1b7-485a-b4d4-b63b54b6db48-kube-api-access-qw6hh\" (UniqueName: \"kubernetes.io/projected/e615817e-f1b7-485a-b4d4-b63b54b6db48-kube-api-access-qw6hh\") pod \"e615817e-f1b7-485a-b4d4-b63b54b6db48\" (UID: \"e615817e-f1b7-485a-b4d4-b63b54b6db48\") "
Oct 31 13:45:23.532006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.531975    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e615817e-f1b7-485a-b4d4-b63b54b6db48-maas-api-service-ca" pod "e615817e-f1b7-485a-b4d4-b63b54b6db48" (UID: "e615817e-f1b7-485a-b4d4-b63b54b6db48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:23.532259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.532237    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07c1925-fc71-432f-9251-1d22de15efd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dc2b72a-hxnsw\" (UID: \"d07c1925-fc71-432f-9251-1d22de15efd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"
Oct 31 13:45:23.532428 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.532402    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07c1925-fc71-432f-9251-1d22de15efd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dc2b72a-hxnsw\" (UID: \"d07c1925-fc71-432f-9251-1d22de15efd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"
Oct 31 13:45:23.532604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.532588    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhzd2\" (UniqueName: \"kubernetes.io/projected/d07c1925-fc71-432f-9251-1d22de15efd0-kube-api-access-vhzd2\") pod \"maas-api-revoke-sub-1dc2b72a-hxnsw\" (UID: \"d07c1925-fc71-432f-9251-1d22de15efd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"
Oct 31 13:45:23.532909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.532886    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e615817e-f1b7-485a-b4d4-b63b54b6db48-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:23.534109 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.534088    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e615817e-f1b7-485a-b4d4-b63b54b6db48-kube-api-access-qw6hh" pod "e615817e-f1b7-485a-b4d4-b63b54b6db48" (UID: "e615817e-f1b7-485a-b4d4-b63b54b6db48"). InnerVolumeSpecName "kube-api-access-qw6hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:23.559803 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:45:23.559772    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e053af8_843e_4184_8a8d_770834fc5cf2.slice/crio-1cf030f0fa3c5d3f44f625694d910c8c060f94856345cef0b4c6f32fd8900c42 WatchSource:0}: Error finding container 1cf030f0fa3c5d3f44f625694d910c8c060f94856345cef0b4c6f32fd8900c42: Status 404 returned error can't find the container with id 1cf030f0fa3c5d3f44f625694d910c8c060f94856345cef0b4c6f32fd8900c42
Oct 31 13:45:23.560252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.560227    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"]
Oct 31 13:45:23.634484 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.634449    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhzd2\" (UniqueName: \"kubernetes.io/projected/d07c1925-fc71-432f-9251-1d22de15efd0-kube-api-access-vhzd2\") pod \"maas-api-revoke-sub-1dc2b72a-hxnsw\" (UID: \"d07c1925-fc71-432f-9251-1d22de15efd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"
Oct 31 13:45:23.634696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.634652    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07c1925-fc71-432f-9251-1d22de15efd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dc2b72a-hxnsw\" (UID: \"d07c1925-fc71-432f-9251-1d22de15efd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"
Oct 31 13:45:23.634784 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.634725    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07c1925-fc71-432f-9251-1d22de15efd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dc2b72a-hxnsw\" (UID: \"d07c1925-fc71-432f-9251-1d22de15efd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"
Oct 31 13:45:23.634854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.634816    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw6hh\" (UniqueName: \"kubernetes.io/projected/e615817e-f1b7-485a-b4d4-b63b54b6db48-kube-api-access-qw6hh\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:23.635332 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.635307    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07c1925-fc71-432f-9251-1d22de15efd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dc2b72a-hxnsw\" (UID: \"d07c1925-fc71-432f-9251-1d22de15efd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"
Oct 31 13:45:23.637432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.637408    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07c1925-fc71-432f-9251-1d22de15efd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dc2b72a-hxnsw\" (UID: \"d07c1925-fc71-432f-9251-1d22de15efd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"
Oct 31 13:45:23.647744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.647720    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhzd2\" (UniqueName: \"kubernetes.io/projected/d07c1925-fc71-432f-9251-1d22de15efd0-kube-api-access-vhzd2\") pod \"maas-api-revoke-sub-1dc2b72a-hxnsw\" (UID: \"d07c1925-fc71-432f-9251-1d22de15efd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"
Oct 31 13:45:23.693969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:23.693942    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"
Oct 31 13:45:24.026434 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:45:24.026403    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07c1925_fc71_432f_9251_1d22de15efd0.slice/crio-6c228c061d2a22aad2e7c8e2935eb949d012f20263ca71562622bc879f2f6b20 WatchSource:0}: Error finding container 6c228c061d2a22aad2e7c8e2935eb949d012f20263ca71562622bc879f2f6b20: Status 404 returned error can't find the container with id 6c228c061d2a22aad2e7c8e2935eb949d012f20263ca71562622bc879f2f6b20
Oct 31 13:45:24.028519 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.028476    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"]
Oct 31 13:45:24.103309 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.103272    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7"
Oct 31 13:45:24.103493 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.103304    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7" event={"ID":"e615817e-f1b7-485a-b4d4-b63b54b6db48","Type":"ContainerDied","Data":"56544e22d36ee98b267c5fb95acbc3f8ea7b27725a4b8f49aaf4d27894dfda03"}
Oct 31 13:45:24.103493 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.103371    4428 scope.go:122] "RemoveContainer" containerID="2ed4a2b116ed36ed6f7475d9869d9dbe350a08e306cf8a3db0c864cdba212013"
Oct 31 13:45:24.109326 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.109284    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4" event={"ID":"3e053af8-843e-4184-8a8d-770834fc5cf2","Type":"ContainerStarted","Data":"ca144209c1fd51b2d8ad611ac02cae7e7a60eaa4bba568bcf087452130845b68"}
Oct 31 13:45:24.109558 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.109355    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4" event={"ID":"3e053af8-843e-4184-8a8d-770834fc5cf2","Type":"ContainerStarted","Data":"1cf030f0fa3c5d3f44f625694d910c8c060f94856345cef0b4c6f32fd8900c42"}
Oct 31 13:45:24.111218 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.111148    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw" event={"ID":"d07c1925-fc71-432f-9251-1d22de15efd0","Type":"ContainerStarted","Data":"6c228c061d2a22aad2e7c8e2935eb949d012f20263ca71562622bc879f2f6b20"}
Oct 31 13:45:24.135023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.134974    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4" podStartSLOduration=2.134960026 podStartE2EDuration="2.134960026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:24.131592291 +0000 UTC m=+2521.529940915" watchObservedRunningTime="2026-10-31 13:45:24.134960026 +0000 UTC m=+2521.533308686"
Oct 31 13:45:24.152565 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.152532    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7"]
Oct 31 13:45:24.156861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.156836    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f4232c-29890905-tz5v7"]
Oct 31 13:45:24.394180 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.392558    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:24.761096 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.761060    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d9f9bc-22ec-4943-8a8c-9ed7abe7af98" path="/var/lib/kubelet/pods/55d9f9bc-22ec-4943-8a8c-9ed7abe7af98/volumes"
Oct 31 13:45:24.761628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.761612    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="680bd885-b3c7-42f5-a3a8-4f2950d0b462" path="/var/lib/kubelet/pods/680bd885-b3c7-42f5-a3a8-4f2950d0b462/volumes"
Oct 31 13:45:24.762063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.762049    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb5549e-6008-46a8-8e55-3b37cb62cd60" path="/var/lib/kubelet/pods/6bb5549e-6008-46a8-8e55-3b37cb62cd60/volumes"
Oct 31 13:45:24.762778 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.762761    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77411480-49b1-49e5-90f0-1fc4416288da" path="/var/lib/kubelet/pods/77411480-49b1-49e5-90f0-1fc4416288da/volumes"
Oct 31 13:45:24.763257 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.763240    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9485106e-58f9-4b42-ae34-0136f6345af2" path="/var/lib/kubelet/pods/9485106e-58f9-4b42-ae34-0136f6345af2/volumes"
Oct 31 13:45:24.763657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.763641    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e615817e-f1b7-485a-b4d4-b63b54b6db48" path="/var/lib/kubelet/pods/e615817e-f1b7-485a-b4d4-b63b54b6db48/volumes"
Oct 31 13:45:24.764437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:24.764419    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e899d7ff-9dd2-48d8-b537-f03e45b6d1c2" path="/var/lib/kubelet/pods/e899d7ff-9dd2-48d8-b537-f03e45b6d1c2/volumes"
Oct 31 13:45:25.128219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.128186    4428 generic.go:356] "Generic (PLEG): container finished" podID="3e053af8-843e-4184-8a8d-770834fc5cf2" containerID="ca144209c1fd51b2d8ad611ac02cae7e7a60eaa4bba568bcf087452130845b68" exitCode=0
Oct 31 13:45:25.128652 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.128265    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4" event={"ID":"3e053af8-843e-4184-8a8d-770834fc5cf2","Type":"ContainerDied","Data":"ca144209c1fd51b2d8ad611ac02cae7e7a60eaa4bba568bcf087452130845b68"}
Oct 31 13:45:25.129958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.129933    4428 generic.go:356] "Generic (PLEG): container finished" podID="d07c1925-fc71-432f-9251-1d22de15efd0" containerID="8f953586f2f032636c9272c50c67dc6765c892ebeccfdb049ccb8070c9efb11a" exitCode=0
Oct 31 13:45:25.130085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.130013    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw" event={"ID":"d07c1925-fc71-432f-9251-1d22de15efd0","Type":"ContainerDied","Data":"8f953586f2f032636c9272c50c67dc6765c892ebeccfdb049ccb8070c9efb11a"}
Oct 31 13:45:25.341778 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.341694    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:25.372215 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.372182    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"]
Oct 31 13:45:25.373116 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.373093    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e615817e-f1b7-485a-b4d4-b63b54b6db48" containerName="cleanup"
Oct 31 13:45:25.373116 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.373112    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e615817e-f1b7-485a-b4d4-b63b54b6db48" containerName="cleanup"
Oct 31 13:45:25.373305 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.373144    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e615817e-f1b7-485a-b4d4-b63b54b6db48" containerName="cleanup"
Oct 31 13:45:25.373305 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.373151    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e615817e-f1b7-485a-b4d4-b63b54b6db48" containerName="cleanup"
Oct 31 13:45:25.373305 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.373282    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e615817e-f1b7-485a-b4d4-b63b54b6db48" containerName="cleanup"
Oct 31 13:45:25.373305 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.373301    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e615817e-f1b7-485a-b4d4-b63b54b6db48" containerName="cleanup"
Oct 31 13:45:25.377197 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.377177    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.387608 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.387576    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"]
Oct 31 13:45:25.388194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.387985    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-72c210\"/\"kube-root-ca.crt\""
Oct 31 13:45:25.388194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.388050    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-72c210\"/\"openshift-service-ca.crt\""
Oct 31 13:45:25.393528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.392218    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-72c210\"/\"default-dockercfg-cfxkf\""
Oct 31 13:45:25.393528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.392794    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-72c210\"/\"test-model-72c210-kserve-self-signed-certs\""
Oct 31 13:45:25.479755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.479723    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe8d194-92d6-471f-8233-15ba5c15b33a-tls-certs\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.480008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.479766    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-dshm\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.480122 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.480066    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-home\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.480207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.480187    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjztz\" (UniqueName: \"kubernetes.io/projected/afe8d194-92d6-471f-8233-15ba5c15b33a-kube-api-access-cjztz\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.480242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.480226    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-tmp-dir\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.480339 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.480322    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-model-cache\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.496197 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:25.496165    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680bd885_b3c7_42f5_a3a8_4f2950d0b462.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice/crio-f4ff7173cf463fa77294cb5afd3e25bd0fbc5e211a501b20b208a215eb151f31\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:45:25.523279 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.523246    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:25.634180 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.634088    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-home\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.756624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.756587    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjztz\" (UniqueName: \"kubernetes.io/projected/afe8d194-92d6-471f-8233-15ba5c15b33a-kube-api-access-cjztz\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.768636 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.762065    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-home\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.770550 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.770520    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-tmp-dir\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.770736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.770575    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-model-cache\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.770736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.770609    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe8d194-92d6-471f-8233-15ba5c15b33a-tls-certs\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.770736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.770648    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-dshm\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.771608 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.771581    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-tmp-dir\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.771847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.771819    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-model-cache\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.798128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.793406    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-dshm\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.800469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.800431    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe8d194-92d6-471f-8233-15ba5c15b33a-tls-certs\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:25.808560 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:25.808532    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjztz\" (UniqueName: \"kubernetes.io/projected/afe8d194-92d6-471f-8233-15ba5c15b33a-kube-api-access-cjztz\") pod \"test-model-72c210-kserve-b4ffdc445-7bvxf\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") " pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:26.002797 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.002756    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:26.308914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.308843    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"]
Oct 31 13:45:26.310513 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:45:26.310488    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafe8d194_92d6_471f_8233_15ba5c15b33a.slice/crio-dc6008af4f35e9e57b069e63adc824b402eaab47a043b7dc3d8632eadd06d755 WatchSource:0}: Error finding container dc6008af4f35e9e57b069e63adc824b402eaab47a043b7dc3d8632eadd06d755: Status 404 returned error can't find the container with id dc6008af4f35e9e57b069e63adc824b402eaab47a043b7dc3d8632eadd06d755
Oct 31 13:45:26.389449 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.389421    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"
Oct 31 13:45:26.394564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.394537    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"
Oct 31 13:45:26.459350 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.459256    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:26.485841 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.485774    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d07c1925-fc71-432f-9251-1d22de15efd0-kube-api-access-vhzd2\" (UniqueName: \"kubernetes.io/projected/d07c1925-fc71-432f-9251-1d22de15efd0-kube-api-access-vhzd2\") pod \"d07c1925-fc71-432f-9251-1d22de15efd0\" (UID: \"d07c1925-fc71-432f-9251-1d22de15efd0\") "
Oct 31 13:45:26.487156 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.487112    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d07c1925-fc71-432f-9251-1d22de15efd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07c1925-fc71-432f-9251-1d22de15efd0-maas-api-xks-ca\") pod \"d07c1925-fc71-432f-9251-1d22de15efd0\" (UID: \"d07c1925-fc71-432f-9251-1d22de15efd0\") "
Oct 31 13:45:26.487646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.487622    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d07c1925-fc71-432f-9251-1d22de15efd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07c1925-fc71-432f-9251-1d22de15efd0-maas-api-openshift-ca\") pod \"d07c1925-fc71-432f-9251-1d22de15efd0\" (UID: \"d07c1925-fc71-432f-9251-1d22de15efd0\") "
Oct 31 13:45:26.489149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.489123    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d07c1925-fc71-432f-9251-1d22de15efd0-maas-api-openshift-ca" pod "d07c1925-fc71-432f-9251-1d22de15efd0" (UID: "d07c1925-fc71-432f-9251-1d22de15efd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:26.568374 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.568221    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07c1925-fc71-432f-9251-1d22de15efd0-kube-api-access-vhzd2" pod "d07c1925-fc71-432f-9251-1d22de15efd0" (UID: "d07c1925-fc71-432f-9251-1d22de15efd0"). InnerVolumeSpecName "kube-api-access-vhzd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:26.568374 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.568361    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d07c1925-fc71-432f-9251-1d22de15efd0-maas-api-xks-ca" pod "d07c1925-fc71-432f-9251-1d22de15efd0" (UID: "d07c1925-fc71-432f-9251-1d22de15efd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:45:26.590892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.589760    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e053af8-843e-4184-8a8d-770834fc5cf2-kube-api-access-cg56z\" (UniqueName: \"kubernetes.io/projected/3e053af8-843e-4184-8a8d-770834fc5cf2-kube-api-access-cg56z\") pod \"3e053af8-843e-4184-8a8d-770834fc5cf2\" (UID: \"3e053af8-843e-4184-8a8d-770834fc5cf2\") "
Oct 31 13:45:26.590892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.589801    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e053af8-843e-4184-8a8d-770834fc5cf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e053af8-843e-4184-8a8d-770834fc5cf2-maas-api-xks-ca\") pod \"3e053af8-843e-4184-8a8d-770834fc5cf2\" (UID: \"3e053af8-843e-4184-8a8d-770834fc5cf2\") "
Oct 31 13:45:26.590892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.590447    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e053af8-843e-4184-8a8d-770834fc5cf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e053af8-843e-4184-8a8d-770834fc5cf2-maas-api-openshift-ca\") pod \"3e053af8-843e-4184-8a8d-770834fc5cf2\" (UID: \"3e053af8-843e-4184-8a8d-770834fc5cf2\") "
Oct 31 13:45:26.592736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.591309    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07c1925-fc71-432f-9251-1d22de15efd0-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:26.592736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.591336    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07c1925-fc71-432f-9251-1d22de15efd0-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:26.592736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.591352    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhzd2\" (UniqueName: \"kubernetes.io/projected/d07c1925-fc71-432f-9251-1d22de15efd0-kube-api-access-vhzd2\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:26.592736 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.591699    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e053af8-843e-4184-8a8d-770834fc5cf2-maas-api-openshift-ca" pod "3e053af8-843e-4184-8a8d-770834fc5cf2" (UID: "3e053af8-843e-4184-8a8d-770834fc5cf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:26.594276 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.593179    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e053af8-843e-4184-8a8d-770834fc5cf2-maas-api-xks-ca" pod "3e053af8-843e-4184-8a8d-770834fc5cf2" (UID: "3e053af8-843e-4184-8a8d-770834fc5cf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:45:26.595387 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.594556    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e053af8-843e-4184-8a8d-770834fc5cf2-kube-api-access-cg56z" pod "3e053af8-843e-4184-8a8d-770834fc5cf2" (UID: "3e053af8-843e-4184-8a8d-770834fc5cf2"). InnerVolumeSpecName "kube-api-access-cg56z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:26.693240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.693209    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e053af8-843e-4184-8a8d-770834fc5cf2-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:26.693240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.693238    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg56z\" (UniqueName: \"kubernetes.io/projected/3e053af8-843e-4184-8a8d-770834fc5cf2-kube-api-access-cg56z\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:26.693485 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:26.693250    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e053af8-843e-4184-8a8d-770834fc5cf2-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:27.151436 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:27.151336    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf" event={"ID":"afe8d194-92d6-471f-8233-15ba5c15b33a","Type":"ContainerStarted","Data":"07718137f5914979f96db589df23fa21d809a92ea9a6f793c2fbc8e436a7d8de"}
Oct 31 13:45:27.151436 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:27.151389    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf" event={"ID":"afe8d194-92d6-471f-8233-15ba5c15b33a","Type":"ContainerStarted","Data":"dc6008af4f35e9e57b069e63adc824b402eaab47a043b7dc3d8632eadd06d755"}
Oct 31 13:45:27.153009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:27.152987    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"
Oct 31 13:45:27.153172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:27.153016    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4" event={"ID":"3e053af8-843e-4184-8a8d-770834fc5cf2","Type":"ContainerDied","Data":"1cf030f0fa3c5d3f44f625694d910c8c060f94856345cef0b4c6f32fd8900c42"}
Oct 31 13:45:27.153172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:27.153060    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cf030f0fa3c5d3f44f625694d910c8c060f94856345cef0b4c6f32fd8900c42"
Oct 31 13:45:27.154700 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:27.154680    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"
Oct 31 13:45:27.154799 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:27.154714    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw" event={"ID":"d07c1925-fc71-432f-9251-1d22de15efd0","Type":"ContainerDied","Data":"6c228c061d2a22aad2e7c8e2935eb949d012f20263ca71562622bc879f2f6b20"}
Oct 31 13:45:27.154799 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:27.154767    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c228c061d2a22aad2e7c8e2935eb949d012f20263ca71562622bc879f2f6b20"
Oct 31 13:45:27.176953 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:27.176902    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf" podStartSLOduration=2.1768885239999998 podStartE2EDuration="2.176888524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:27.170986134 +0000 UTC m=+2524.569334759" watchObservedRunningTime="2026-10-31 13:45:27.176888524 +0000 UTC m=+2524.575237145"
Oct 31 13:45:27.412951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:27.412525    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:28.003560 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:28.003528    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:28.221892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:28.221468    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:28.855176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:28.855144    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"
Oct 31 13:45:28.855639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:28.855589    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"
Oct 31 13:45:29.220635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:29.217835    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:30.123893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:30.120068    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:30.345738 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:30.345685    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680bd885_b3c7_42f5_a3a8_4f2950d0b462.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice/crio-f4ff7173cf463fa77294cb5afd3e25bd0fbc5e211a501b20b208a215eb151f31\": RecentStats: unable to find data in memory cache]"
Oct 31 13:45:30.940946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:30.940909    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:31.330528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:31.330489    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"]
Oct 31 13:45:31.364911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:31.364862    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"]
Oct 31 13:45:31.496496 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:31.496456    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:31.670172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:31.669999    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:32.003226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:32.003187    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:32.998173 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:32.998133    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:33.896493 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:33.896458    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:34.069844 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.063486    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:34.387726 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.387637    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"]
Oct 31 13:45:34.388526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.388508    4428 cpu_manager.go:414] "containerMap: removing container" podUID="3e053af8-843e-4184-8a8d-770834fc5cf2" containerName="revoke-keys"
Oct 31 13:45:34.388526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.388526    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e053af8-843e-4184-8a8d-770834fc5cf2" containerName="revoke-keys"
Oct 31 13:45:34.388664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.388538    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d07c1925-fc71-432f-9251-1d22de15efd0" containerName="revoke-keys"
Oct 31 13:45:34.388664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.388543    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d07c1925-fc71-432f-9251-1d22de15efd0" containerName="revoke-keys"
Oct 31 13:45:34.388664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.388594    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e615817e-f1b7-485a-b4d4-b63b54b6db48" containerName="cleanup"
Oct 31 13:45:34.388664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.388600    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e615817e-f1b7-485a-b4d4-b63b54b6db48" containerName="cleanup"
Oct 31 13:45:34.388798 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.388728    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e615817e-f1b7-485a-b4d4-b63b54b6db48" containerName="cleanup"
Oct 31 13:45:34.388798 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.388740    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e053af8-843e-4184-8a8d-770834fc5cf2" containerName="revoke-keys"
Oct 31 13:45:34.388798 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.388746    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d07c1925-fc71-432f-9251-1d22de15efd0" containerName="revoke-keys"
Oct 31 13:45:34.395738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.395705    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"
Oct 31 13:45:34.404922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.404111    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:45:34.404922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.404114    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:45:34.404922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.404566    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"]
Oct 31 13:45:34.498826 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.498792    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t2lx\" (UniqueName: \"kubernetes.io/projected/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-kube-api-access-9t2lx\") pod \"maas-api-revoke-sub-d1585303-8lskb\" (UID: \"2da62d96-c7bc-4502-b7a2-4ea716efbe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"
Oct 31 13:45:34.499073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.499057    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1585303-8lskb\" (UID: \"2da62d96-c7bc-4502-b7a2-4ea716efbe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"
Oct 31 13:45:34.499135 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.499118    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1585303-8lskb\" (UID: \"2da62d96-c7bc-4502-b7a2-4ea716efbe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"
Oct 31 13:45:34.600712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.600673    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1585303-8lskb\" (UID: \"2da62d96-c7bc-4502-b7a2-4ea716efbe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"
Oct 31 13:45:34.600914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.600754    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1585303-8lskb\" (UID: \"2da62d96-c7bc-4502-b7a2-4ea716efbe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"
Oct 31 13:45:34.600914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.600841    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t2lx\" (UniqueName: \"kubernetes.io/projected/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-kube-api-access-9t2lx\") pod \"maas-api-revoke-sub-d1585303-8lskb\" (UID: \"2da62d96-c7bc-4502-b7a2-4ea716efbe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"
Oct 31 13:45:34.601570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.601497    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1585303-8lskb\" (UID: \"2da62d96-c7bc-4502-b7a2-4ea716efbe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"
Oct 31 13:45:34.603284 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.603263    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1585303-8lskb\" (UID: \"2da62d96-c7bc-4502-b7a2-4ea716efbe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"
Oct 31 13:45:34.618788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.618765    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t2lx\" (UniqueName: \"kubernetes.io/projected/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-kube-api-access-9t2lx\") pod \"maas-api-revoke-sub-d1585303-8lskb\" (UID: \"2da62d96-c7bc-4502-b7a2-4ea716efbe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"
Oct 31 13:45:34.727025 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.726990    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"
Oct 31 13:45:34.781921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.780631    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"]
Oct 31 13:45:34.782493 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.782426    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h" podUID="511e4897-ed7c-463b-a199-1e48519db98a" containerName="maas-api" containerID="cri-o://3a5ddf61d140a274fb85057f0ff398ad997958bd8982717a9339a3784fa6212f" gracePeriod=30
Oct 31 13:45:34.892363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:34.892293    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:35.171892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.149778    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:35.253949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.194625    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"]
Oct 31 13:45:35.329054 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.329012    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb" event={"ID":"2da62d96-c7bc-4502-b7a2-4ea716efbe2e","Type":"ContainerStarted","Data":"660b8d333032f942a151f0a463883a536617951d3545c700824c47fb400784c7"}
Oct 31 13:45:35.338896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.336073    4428 generic.go:356] "Generic (PLEG): container finished" podID="511e4897-ed7c-463b-a199-1e48519db98a" containerID="3a5ddf61d140a274fb85057f0ff398ad997958bd8982717a9339a3784fa6212f" exitCode=0
Oct 31 13:45:35.338896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.336258    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h" event={"ID":"511e4897-ed7c-463b-a199-1e48519db98a","Type":"ContainerDied","Data":"3a5ddf61d140a274fb85057f0ff398ad997958bd8982717a9339a3784fa6212f"}
Oct 31 13:45:35.453173 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.453140    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:35.463282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.463255    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:45:35.519226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.519136    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/511e4897-ed7c-463b-a199-1e48519db98a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/511e4897-ed7c-463b-a199-1e48519db98a-openshift-service-ca\") pod \"511e4897-ed7c-463b-a199-1e48519db98a\" (UID: \"511e4897-ed7c-463b-a199-1e48519db98a\") "
Oct 31 13:45:35.519409 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.519285    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/511e4897-ed7c-463b-a199-1e48519db98a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/511e4897-ed7c-463b-a199-1e48519db98a-metrics-certs\") pod \"511e4897-ed7c-463b-a199-1e48519db98a\" (UID: \"511e4897-ed7c-463b-a199-1e48519db98a\") "
Oct 31 13:45:35.519409 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.519308    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/511e4897-ed7c-463b-a199-1e48519db98a-kube-api-access-rsx7c\" (UniqueName: \"kubernetes.io/projected/511e4897-ed7c-463b-a199-1e48519db98a-kube-api-access-rsx7c\") pod \"511e4897-ed7c-463b-a199-1e48519db98a\" (UID: \"511e4897-ed7c-463b-a199-1e48519db98a\") "
Oct 31 13:45:35.519409 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.519341    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/511e4897-ed7c-463b-a199-1e48519db98a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/511e4897-ed7c-463b-a199-1e48519db98a-maas-api-tls\") pod \"511e4897-ed7c-463b-a199-1e48519db98a\" (UID: \"511e4897-ed7c-463b-a199-1e48519db98a\") "
Oct 31 13:45:35.519622 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.519541    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511e4897-ed7c-463b-a199-1e48519db98a-openshift-service-ca" pod "511e4897-ed7c-463b-a199-1e48519db98a" (UID: "511e4897-ed7c-463b-a199-1e48519db98a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:35.520170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.520123    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/511e4897-ed7c-463b-a199-1e48519db98a-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:35.521588 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.521561    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511e4897-ed7c-463b-a199-1e48519db98a-metrics-certs" pod "511e4897-ed7c-463b-a199-1e48519db98a" (UID: "511e4897-ed7c-463b-a199-1e48519db98a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:45:35.522204 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.522180    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511e4897-ed7c-463b-a199-1e48519db98a-maas-api-tls" pod "511e4897-ed7c-463b-a199-1e48519db98a" (UID: "511e4897-ed7c-463b-a199-1e48519db98a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:45:35.522320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.522295    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511e4897-ed7c-463b-a199-1e48519db98a-kube-api-access-rsx7c" pod "511e4897-ed7c-463b-a199-1e48519db98a" (UID: "511e4897-ed7c-463b-a199-1e48519db98a"). InnerVolumeSpecName "kube-api-access-rsx7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:35.653999 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.653906    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:35.677064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.657276    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/511e4897-ed7c-463b-a199-1e48519db98a-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:35.677064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.657316    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsx7c\" (UniqueName: \"kubernetes.io/projected/511e4897-ed7c-463b-a199-1e48519db98a-kube-api-access-rsx7c\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:35.677064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.657333    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/511e4897-ed7c-463b-a199-1e48519db98a-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:35.911088 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:35.911039    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:36.003009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:36.002956    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:36.016603 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:36.016546    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:36.331798 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:36.331758    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n" podUID="1769894b-3274-4280-b0e2-a7056d77a96a" containerName="payload-processing" containerID="cri-o://d4fe255e47338f623d48d69d838488b47d653bb8499600f7cc8047a29e178032" gracePeriod=25
Oct 31 13:45:36.347229 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:36.347177    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb" event={"ID":"2da62d96-c7bc-4502-b7a2-4ea716efbe2e","Type":"ContainerStarted","Data":"4902afdd2cfbe96d232b61d5e453eb846ee24e8aedc8b4b370f53b008845977e"}
Oct 31 13:45:36.350000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:36.349968    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"
Oct 31 13:45:36.350000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:36.350018    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h" event={"ID":"511e4897-ed7c-463b-a199-1e48519db98a","Type":"ContainerDied","Data":"527f89ddaff3b1b7f6d5c9bfb9b2cab2c3d5dd6c1987f738a65a73de3cbabdff"}
Oct 31 13:45:36.350000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:36.350069    4428 scope.go:122] "RemoveContainer" containerID="3a5ddf61d140a274fb85057f0ff398ad997958bd8982717a9339a3784fa6212f"
Oct 31 13:45:36.372833 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:36.372761    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg" podUID="e5d14076-4049-4177-9a99-738b8d8e808b" containerName="payload-pre-processing" containerID="cri-o://dfbd850fd86091acd667d73c628781f3e3de84147b5fe30ebdf4e6a57c239648" gracePeriod=25
Oct 31 13:45:36.375363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:36.375127    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb" podStartSLOduration=2.375109013 podStartE2EDuration="2.375109013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:36.369254331 +0000 UTC m=+2533.767602957" watchObservedRunningTime="2026-10-31 13:45:36.375109013 +0000 UTC m=+2533.773457639"
Oct 31 13:45:36.392028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:36.391858    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:45:36.410669 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:36.410522    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"]
Oct 31 13:45:36.422651 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:36.422614    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h"]
Oct 31 13:45:36.480490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:36.479492    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:36.818985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:36.817977    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="511e4897-ed7c-463b-a199-1e48519db98a" path="/var/lib/kubelet/pods/511e4897-ed7c-463b-a199-1e48519db98a/volumes"
Oct 31 13:45:36.937305 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:36.937258    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:37.029045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.029005    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"]
Oct 31 13:45:37.031615 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.030801    4428 cpu_manager.go:414] "containerMap: removing container" podUID="511e4897-ed7c-463b-a199-1e48519db98a" containerName="maas-api"
Oct 31 13:45:37.037160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.037128    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="511e4897-ed7c-463b-a199-1e48519db98a" containerName="maas-api"
Oct 31 13:45:37.039102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.039074    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="511e4897-ed7c-463b-a199-1e48519db98a" containerName="maas-api"
Oct 31 13:45:37.057625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.057597    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"
Oct 31 13:45:37.085710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.085631    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"]
Oct 31 13:45:37.210068 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.210033    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwd58\" (UniqueName: \"kubernetes.io/projected/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-kube-api-access-cwd58\") pod \"maas-api-revoke-sub-0d28499f-qqsbz\" (UID: \"a06d3d41-b562-4a1a-8dea-3fabee0fb40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"
Oct 31 13:45:37.210238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.210077    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d28499f-qqsbz\" (UID: \"a06d3d41-b562-4a1a-8dea-3fabee0fb40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"
Oct 31 13:45:37.210238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.210103    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d28499f-qqsbz\" (UID: \"a06d3d41-b562-4a1a-8dea-3fabee0fb40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"
Oct 31 13:45:37.311706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.311673    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwd58\" (UniqueName: \"kubernetes.io/projected/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-kube-api-access-cwd58\") pod \"maas-api-revoke-sub-0d28499f-qqsbz\" (UID: \"a06d3d41-b562-4a1a-8dea-3fabee0fb40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"
Oct 31 13:45:37.311706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.311713    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d28499f-qqsbz\" (UID: \"a06d3d41-b562-4a1a-8dea-3fabee0fb40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"
Oct 31 13:45:37.311968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.311737    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d28499f-qqsbz\" (UID: \"a06d3d41-b562-4a1a-8dea-3fabee0fb40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"
Oct 31 13:45:37.312431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.312411    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d28499f-qqsbz\" (UID: \"a06d3d41-b562-4a1a-8dea-3fabee0fb40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"
Oct 31 13:45:37.314555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.314524    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d28499f-qqsbz\" (UID: \"a06d3d41-b562-4a1a-8dea-3fabee0fb40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"
Oct 31 13:45:37.327416 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.327382    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwd58\" (UniqueName: \"kubernetes.io/projected/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-kube-api-access-cwd58\") pod \"maas-api-revoke-sub-0d28499f-qqsbz\" (UID: \"a06d3d41-b562-4a1a-8dea-3fabee0fb40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"
Oct 31 13:45:37.361622 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.361538    4428 generic.go:356] "Generic (PLEG): container finished" podID="2da62d96-c7bc-4502-b7a2-4ea716efbe2e" containerID="4902afdd2cfbe96d232b61d5e453eb846ee24e8aedc8b4b370f53b008845977e" exitCode=0
Oct 31 13:45:37.362097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.361619    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb" event={"ID":"2da62d96-c7bc-4502-b7a2-4ea716efbe2e","Type":"ContainerDied","Data":"4902afdd2cfbe96d232b61d5e453eb846ee24e8aedc8b4b370f53b008845977e"}
Oct 31 13:45:37.412238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.412204    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"
Oct 31 13:45:37.506298 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.506261    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg" podUID="e5d14076-4049-4177-9a99-738b8d8e808b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:45:37.710910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.710850    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"]
Oct 31 13:45:37.712088 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:45:37.712060    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06d3d41_b562_4a1a_8dea_3fabee0fb40c.slice/crio-13c850d59c85ce3977b72b5afede8f743ce8acb494a0c6944f71ed4c6952a540 WatchSource:0}: Error finding container 13c850d59c85ce3977b72b5afede8f743ce8acb494a0c6944f71ed4c6952a540: Status 404 returned error can't find the container with id 13c850d59c85ce3977b72b5afede8f743ce8acb494a0c6944f71ed4c6952a540
Oct 31 13:45:37.926894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:37.925966    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:38.374008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:38.373916    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz" event={"ID":"a06d3d41-b562-4a1a-8dea-3fabee0fb40c","Type":"ContainerStarted","Data":"fd75ebb9237485312da43ce357e10a4a8f559227d30cd80145ce663149634252"}
Oct 31 13:45:38.374008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:38.373980    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz" event={"ID":"a06d3d41-b562-4a1a-8dea-3fabee0fb40c","Type":"ContainerStarted","Data":"13c850d59c85ce3977b72b5afede8f743ce8acb494a0c6944f71ed4c6952a540"}
Oct 31 13:45:38.400814 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:38.400743    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz" podStartSLOduration=2.400723117 podStartE2EDuration="2.400723117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:38.393742698 +0000 UTC m=+2535.792091322" watchObservedRunningTime="2026-10-31 13:45:38.400723117 +0000 UTC m=+2535.799071743"
Oct 31 13:45:38.432072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:38.432014    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb50fa-5746f7f488-wzc4h" podUID="511e4897-ed7c-463b-a199-1e48519db98a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.189:8443/health\": context deadline exceeded"
Oct 31 13:45:38.551013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:38.550986    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"
Oct 31 13:45:38.737273 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:38.737232    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-kube-api-access-9t2lx\" (UniqueName: \"kubernetes.io/projected/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-kube-api-access-9t2lx\") pod \"2da62d96-c7bc-4502-b7a2-4ea716efbe2e\" (UID: \"2da62d96-c7bc-4502-b7a2-4ea716efbe2e\") "
Oct 31 13:45:38.737588 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:38.737567    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-maas-api-xks-ca\") pod \"2da62d96-c7bc-4502-b7a2-4ea716efbe2e\" (UID: \"2da62d96-c7bc-4502-b7a2-4ea716efbe2e\") "
Oct 31 13:45:38.737718 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:38.737698    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-maas-api-openshift-ca\") pod \"2da62d96-c7bc-4502-b7a2-4ea716efbe2e\" (UID: \"2da62d96-c7bc-4502-b7a2-4ea716efbe2e\") "
Oct 31 13:45:38.738100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:38.738076    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-maas-api-openshift-ca" pod "2da62d96-c7bc-4502-b7a2-4ea716efbe2e" (UID: "2da62d96-c7bc-4502-b7a2-4ea716efbe2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:38.738427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:38.738403    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:38.739730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:38.739702    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-kube-api-access-9t2lx" pod "2da62d96-c7bc-4502-b7a2-4ea716efbe2e" (UID: "2da62d96-c7bc-4502-b7a2-4ea716efbe2e"). InnerVolumeSpecName "kube-api-access-9t2lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:38.739923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:38.739901    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-maas-api-xks-ca" pod "2da62d96-c7bc-4502-b7a2-4ea716efbe2e" (UID: "2da62d96-c7bc-4502-b7a2-4ea716efbe2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:45:38.809745 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:38.809683    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n" podUID="1769894b-3274-4280-b0e2-a7056d77a96a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:45:38.845901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:38.844692    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:38.845901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:38.844772    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t2lx\" (UniqueName: \"kubernetes.io/projected/2da62d96-c7bc-4502-b7a2-4ea716efbe2e-kube-api-access-9t2lx\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:38.859775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:38.853917    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:39.384291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:39.384256    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"
Oct 31 13:45:39.384727 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:39.384261    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb" event={"ID":"2da62d96-c7bc-4502-b7a2-4ea716efbe2e","Type":"ContainerDied","Data":"660b8d333032f942a151f0a463883a536617951d3545c700824c47fb400784c7"}
Oct 31 13:45:39.384727 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:39.384429    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="660b8d333032f942a151f0a463883a536617951d3545c700824c47fb400784c7"
Oct 31 13:45:39.386172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:39.386113    4428 generic.go:356] "Generic (PLEG): container finished" podID="a06d3d41-b562-4a1a-8dea-3fabee0fb40c" containerID="fd75ebb9237485312da43ce357e10a4a8f559227d30cd80145ce663149634252" exitCode=0
Oct 31 13:45:39.386314 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:39.386211    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz" event={"ID":"a06d3d41-b562-4a1a-8dea-3fabee0fb40c","Type":"ContainerDied","Data":"fd75ebb9237485312da43ce357e10a4a8f559227d30cd80145ce663149634252"}
Oct 31 13:45:39.703946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:39.699500    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:40.177805 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.177763    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:40.481265 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:40.481219    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice/crio-f4ff7173cf463fa77294cb5afd3e25bd0fbc5e211a501b20b208a215eb151f31\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680bd885_b3c7_42f5_a3a8_4f2950d0b462.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:45:40.481681 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:40.481538    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice/crio-f4ff7173cf463fa77294cb5afd3e25bd0fbc5e211a501b20b208a215eb151f31\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680bd885_b3c7_42f5_a3a8_4f2950d0b462.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:45:40.651168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.651141    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"
Oct 31 13:45:40.662978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.662951    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"]
Oct 31 13:45:40.663785 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.663770    4428 cpu_manager.go:414] "containerMap: removing container" podUID="a06d3d41-b562-4a1a-8dea-3fabee0fb40c" containerName="revoke-keys"
Oct 31 13:45:40.663843 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.663787    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a06d3d41-b562-4a1a-8dea-3fabee0fb40c" containerName="revoke-keys"
Oct 31 13:45:40.663843 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.663814    4428 cpu_manager.go:414] "containerMap: removing container" podUID="2da62d96-c7bc-4502-b7a2-4ea716efbe2e" containerName="revoke-keys"
Oct 31 13:45:40.663843 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.663819    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da62d96-c7bc-4502-b7a2-4ea716efbe2e" containerName="revoke-keys"
Oct 31 13:45:40.663987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.663974    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a06d3d41-b562-4a1a-8dea-3fabee0fb40c" containerName="revoke-keys"
Oct 31 13:45:40.664035 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.663997    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da62d96-c7bc-4502-b7a2-4ea716efbe2e" containerName="revoke-keys"
Oct 31 13:45:40.670562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.670535    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"
Oct 31 13:45:40.681223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.681148    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"]
Oct 31 13:45:40.692586 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.692555    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-maas-api-openshift-ca\") pod \"a06d3d41-b562-4a1a-8dea-3fabee0fb40c\" (UID: \"a06d3d41-b562-4a1a-8dea-3fabee0fb40c\") "
Oct 31 13:45:40.692756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.692693    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-maas-api-xks-ca\") pod \"a06d3d41-b562-4a1a-8dea-3fabee0fb40c\" (UID: \"a06d3d41-b562-4a1a-8dea-3fabee0fb40c\") "
Oct 31 13:45:40.692841 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.692824    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-kube-api-access-cwd58\" (UniqueName: \"kubernetes.io/projected/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-kube-api-access-cwd58\") pod \"a06d3d41-b562-4a1a-8dea-3fabee0fb40c\" (UID: \"a06d3d41-b562-4a1a-8dea-3fabee0fb40c\") "
Oct 31 13:45:40.693416 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.693387    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjvkf\" (UniqueName: \"kubernetes.io/projected/5cd48437-2d0c-485f-952d-e5116c9514da-kube-api-access-vjvkf\") pod \"maas-api-revoke-sub-6b5a3559-fbrl5\" (UID: \"5cd48437-2d0c-485f-952d-e5116c9514da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"
Oct 31 13:45:40.693779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.693527    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd48437-2d0c-485f-952d-e5116c9514da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b5a3559-fbrl5\" (UID: \"5cd48437-2d0c-485f-952d-e5116c9514da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"
Oct 31 13:45:40.693899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.693851    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd48437-2d0c-485f-952d-e5116c9514da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b5a3559-fbrl5\" (UID: \"5cd48437-2d0c-485f-952d-e5116c9514da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"
Oct 31 13:45:40.694264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.694239    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-maas-api-openshift-ca" pod "a06d3d41-b562-4a1a-8dea-3fabee0fb40c" (UID: "a06d3d41-b562-4a1a-8dea-3fabee0fb40c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:40.696105 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.696064    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-kube-api-access-cwd58" pod "a06d3d41-b562-4a1a-8dea-3fabee0fb40c" (UID: "a06d3d41-b562-4a1a-8dea-3fabee0fb40c"). InnerVolumeSpecName "kube-api-access-cwd58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:40.699158 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.699132    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-maas-api-xks-ca" pod "a06d3d41-b562-4a1a-8dea-3fabee0fb40c" (UID: "a06d3d41-b562-4a1a-8dea-3fabee0fb40c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:45:40.829893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.829290    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd48437-2d0c-485f-952d-e5116c9514da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b5a3559-fbrl5\" (UID: \"5cd48437-2d0c-485f-952d-e5116c9514da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"
Oct 31 13:45:40.829893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.829498    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjvkf\" (UniqueName: \"kubernetes.io/projected/5cd48437-2d0c-485f-952d-e5116c9514da-kube-api-access-vjvkf\") pod \"maas-api-revoke-sub-6b5a3559-fbrl5\" (UID: \"5cd48437-2d0c-485f-952d-e5116c9514da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"
Oct 31 13:45:40.829893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.829619    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd48437-2d0c-485f-952d-e5116c9514da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b5a3559-fbrl5\" (UID: \"5cd48437-2d0c-485f-952d-e5116c9514da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"
Oct 31 13:45:40.829893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.829855    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:40.830281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.829960    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:40.830281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.829977    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwd58\" (UniqueName: \"kubernetes.io/projected/a06d3d41-b562-4a1a-8dea-3fabee0fb40c-kube-api-access-cwd58\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:40.838991 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.838488    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd48437-2d0c-485f-952d-e5116c9514da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b5a3559-fbrl5\" (UID: \"5cd48437-2d0c-485f-952d-e5116c9514da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"
Oct 31 13:45:40.850914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.845801    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd48437-2d0c-485f-952d-e5116c9514da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b5a3559-fbrl5\" (UID: \"5cd48437-2d0c-485f-952d-e5116c9514da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"
Oct 31 13:45:40.850914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.848019    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjvkf\" (UniqueName: \"kubernetes.io/projected/5cd48437-2d0c-485f-952d-e5116c9514da-kube-api-access-vjvkf\") pod \"maas-api-revoke-sub-6b5a3559-fbrl5\" (UID: \"5cd48437-2d0c-485f-952d-e5116c9514da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"
Oct 31 13:45:40.878854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.878805    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:40.997135 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:40.997103    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"
Oct 31 13:45:41.295990 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:45:41.295959    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd48437_2d0c_485f_952d_e5116c9514da.slice/crio-4612a42043aba0bab38604d4d1523730f5d8077b69259bb9e63c8881a422a076 WatchSource:0}: Error finding container 4612a42043aba0bab38604d4d1523730f5d8077b69259bb9e63c8881a422a076: Status 404 returned error can't find the container with id 4612a42043aba0bab38604d4d1523730f5d8077b69259bb9e63c8881a422a076
Oct 31 13:45:41.296908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:41.296820    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"]
Oct 31 13:45:41.438371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:41.438330    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5" event={"ID":"5cd48437-2d0c-485f-952d-e5116c9514da","Type":"ContainerStarted","Data":"4612a42043aba0bab38604d4d1523730f5d8077b69259bb9e63c8881a422a076"}
Oct 31 13:45:41.440041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:41.440007    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"
Oct 31 13:45:41.440278 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:41.440031    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz" event={"ID":"a06d3d41-b562-4a1a-8dea-3fabee0fb40c","Type":"ContainerDied","Data":"13c850d59c85ce3977b72b5afede8f743ce8acb494a0c6944f71ed4c6952a540"}
Oct 31 13:45:41.440350 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:41.440314    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13c850d59c85ce3977b72b5afede8f743ce8acb494a0c6944f71ed4c6952a540"
Oct 31 13:45:41.873466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:41.873099    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:42.450631 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:42.450594    4428 generic.go:356] "Generic (PLEG): container finished" podID="5cd48437-2d0c-485f-952d-e5116c9514da" containerID="c6700bc7f1a353e6cefc34785d2baddcb864a323ef2a062daee3b9debc6d9ddd" exitCode=0
Oct 31 13:45:42.450812 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:42.450637    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5" event={"ID":"5cd48437-2d0c-485f-952d-e5116c9514da","Type":"ContainerDied","Data":"c6700bc7f1a353e6cefc34785d2baddcb864a323ef2a062daee3b9debc6d9ddd"}
Oct 31 13:45:42.810325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:42.810291    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"]
Oct 31 13:45:42.815377 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:42.815351    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:45:42.823929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:42.823896    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-6a3533-serving-cert\""
Oct 31 13:45:42.829791 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:42.829099    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"]
Oct 31 13:45:42.918260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:42.918227    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:42.978888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:42.973599    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggj99\" (UniqueName: \"kubernetes.io/projected/b844988a-dac6-463f-898b-ec37e3447ff1-kube-api-access-ggj99\") pod \"maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l\" (UID: \"b844988a-dac6-463f-898b-ec37e3447ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:45:42.978888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:42.973902    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b844988a-dac6-463f-898b-ec37e3447ff1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l\" (UID: \"b844988a-dac6-463f-898b-ec37e3447ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:45:42.978888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:42.973933    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b844988a-dac6-463f-898b-ec37e3447ff1-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l\" (UID: \"b844988a-dac6-463f-898b-ec37e3447ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:45:42.978888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:42.973977    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b844988a-dac6-463f-898b-ec37e3447ff1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l\" (UID: \"b844988a-dac6-463f-898b-ec37e3447ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:45:43.075921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.075818    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b844988a-dac6-463f-898b-ec37e3447ff1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l\" (UID: \"b844988a-dac6-463f-898b-ec37e3447ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:45:43.075921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.075852    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b844988a-dac6-463f-898b-ec37e3447ff1-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l\" (UID: \"b844988a-dac6-463f-898b-ec37e3447ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:45:43.075921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.075917    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b844988a-dac6-463f-898b-ec37e3447ff1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l\" (UID: \"b844988a-dac6-463f-898b-ec37e3447ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:45:43.076210 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.076076    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggj99\" (UniqueName: \"kubernetes.io/projected/b844988a-dac6-463f-898b-ec37e3447ff1-kube-api-access-ggj99\") pod \"maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l\" (UID: \"b844988a-dac6-463f-898b-ec37e3447ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:45:43.076701 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.076669    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b844988a-dac6-463f-898b-ec37e3447ff1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l\" (UID: \"b844988a-dac6-463f-898b-ec37e3447ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:45:43.078585 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.078560    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b844988a-dac6-463f-898b-ec37e3447ff1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l\" (UID: \"b844988a-dac6-463f-898b-ec37e3447ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:45:43.078681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.078584    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b844988a-dac6-463f-898b-ec37e3447ff1-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l\" (UID: \"b844988a-dac6-463f-898b-ec37e3447ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:45:43.088431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.088409    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:43.090034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.090016    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggj99\" (UniqueName: \"kubernetes.io/projected/b844988a-dac6-463f-898b-ec37e3447ff1-kube-api-access-ggj99\") pod \"maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l\" (UID: \"b844988a-dac6-463f-898b-ec37e3447ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:45:43.147562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.147531    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:45:43.247331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.247300    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:43.496139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.494614    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"]
Oct 31 13:45:43.795710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.795682    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"
Oct 31 13:45:43.828504 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.828468    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd48437-2d0c-485f-952d-e5116c9514da-kube-api-access-vjvkf\" (UniqueName: \"kubernetes.io/projected/5cd48437-2d0c-485f-952d-e5116c9514da-kube-api-access-vjvkf\") pod \"5cd48437-2d0c-485f-952d-e5116c9514da\" (UID: \"5cd48437-2d0c-485f-952d-e5116c9514da\") "
Oct 31 13:45:43.828778 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.828600    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd48437-2d0c-485f-952d-e5116c9514da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd48437-2d0c-485f-952d-e5116c9514da-maas-api-openshift-ca\") pod \"5cd48437-2d0c-485f-952d-e5116c9514da\" (UID: \"5cd48437-2d0c-485f-952d-e5116c9514da\") "
Oct 31 13:45:43.828860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.828786    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cd48437-2d0c-485f-952d-e5116c9514da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd48437-2d0c-485f-952d-e5116c9514da-maas-api-xks-ca\") pod \"5cd48437-2d0c-485f-952d-e5116c9514da\" (UID: \"5cd48437-2d0c-485f-952d-e5116c9514da\") "
Oct 31 13:45:43.829068 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.829037    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd48437-2d0c-485f-952d-e5116c9514da-maas-api-openshift-ca" pod "5cd48437-2d0c-485f-952d-e5116c9514da" (UID: "5cd48437-2d0c-485f-952d-e5116c9514da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:43.830350 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.830323    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd48437-2d0c-485f-952d-e5116c9514da-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:43.831558 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.831524    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd48437-2d0c-485f-952d-e5116c9514da-kube-api-access-vjvkf" pod "5cd48437-2d0c-485f-952d-e5116c9514da" (UID: "5cd48437-2d0c-485f-952d-e5116c9514da"). InnerVolumeSpecName "kube-api-access-vjvkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:43.831957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.831937    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd48437-2d0c-485f-952d-e5116c9514da-maas-api-xks-ca" pod "5cd48437-2d0c-485f-952d-e5116c9514da" (UID: "5cd48437-2d0c-485f-952d-e5116c9514da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:45:43.931684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.931646    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd48437-2d0c-485f-952d-e5116c9514da-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:43.931684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:43.931680    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjvkf\" (UniqueName: \"kubernetes.io/projected/5cd48437-2d0c-485f-952d-e5116c9514da-kube-api-access-vjvkf\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:44.129207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:44.128707    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:44.399231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:44.399143    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:44.585944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:44.585884    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:44.812890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:44.809223    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:44.846897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:44.845929    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"
Oct 31 13:45:44.860890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:44.853267    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:45:44.860890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:44.853319    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5" event={"ID":"5cd48437-2d0c-485f-952d-e5116c9514da","Type":"ContainerDied","Data":"4612a42043aba0bab38604d4d1523730f5d8077b69259bb9e63c8881a422a076"}
Oct 31 13:45:44.860890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:44.853351    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4612a42043aba0bab38604d4d1523730f5d8077b69259bb9e63c8881a422a076"
Oct 31 13:45:44.860890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:44.853364    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l" event={"ID":"b844988a-dac6-463f-898b-ec37e3447ff1","Type":"ContainerStarted","Data":"2018b06c0d8e27bd42a7be51b1f078af63bc98a8f2984ada65e440e02c27fb02"}
Oct 31 13:45:44.860890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:44.853386    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"]
Oct 31 13:45:44.860890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:44.853420    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l" event={"ID":"b844988a-dac6-463f-898b-ec37e3447ff1","Type":"ContainerStarted","Data":"2dc3f0be25e79c64cbb6cfa9c4e05950cd23f2c85335e714e082f9a0c68b176d"}
Oct 31 13:45:44.860890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:44.853589    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy" containerID="cri-o://04041c2d8e8caaea1b2dfbe3c3d2578a27ca932badca2417c101e56c6cfbea02" gracePeriod=30
Oct 31 13:45:44.912945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:44.907821    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l" podStartSLOduration=2.363926459 podStartE2EDuration="2.907800687s" totalImagesPullingTime="543.874228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:44.881288156 +0000 UTC m=+2542.279636833" watchObservedRunningTime="2026-10-31 13:45:44.907800687 +0000 UTC m=+2542.306149312"
Oct 31 13:45:45.296449 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.296412    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"]
Oct 31 13:45:45.297394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.297375    4428 cpu_manager.go:414] "containerMap: removing container" podUID="5cd48437-2d0c-485f-952d-e5116c9514da" containerName="revoke-keys"
Oct 31 13:45:45.297494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.297398    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd48437-2d0c-485f-952d-e5116c9514da" containerName="revoke-keys"
Oct 31 13:45:45.297605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.297594    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd48437-2d0c-485f-952d-e5116c9514da" containerName="revoke-keys"
Oct 31 13:45:45.301900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.301881    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"
Oct 31 13:45:45.310114 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.310084    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"]
Oct 31 13:45:45.326627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.326121    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:45:45.326627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.326121    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:45:45.362329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.362269    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95prk\" (UniqueName: \"kubernetes.io/projected/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-kube-api-access-95prk\") pod \"maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6\" (UID: \"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"
Oct 31 13:45:45.362513 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.362423    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6\" (UID: \"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"
Oct 31 13:45:45.362650 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.362630    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6\" (UID: \"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"
Oct 31 13:45:45.464722 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.464685    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6\" (UID: \"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"
Oct 31 13:45:45.464971 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.464838    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95prk\" (UniqueName: \"kubernetes.io/projected/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-kube-api-access-95prk\") pod \"maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6\" (UID: \"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"
Oct 31 13:45:45.464971 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.464902    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6\" (UID: \"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"
Oct 31 13:45:45.465570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.465543    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6\" (UID: \"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"
Oct 31 13:45:45.467487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.467467    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6\" (UID: \"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"
Oct 31 13:45:45.479040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.479019    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95prk\" (UniqueName: \"kubernetes.io/projected/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-kube-api-access-95prk\") pod \"maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6\" (UID: \"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"
Oct 31 13:45:45.522742 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.522702    4428 patch_prober.go:29] interesting pod/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:45:45.522920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.522763    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:45:45.621214 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.621137    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"
Oct 31 13:45:45.804825 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.804769    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:45.919524 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:45:45.919489    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd770fd_6968_41a0_ab2d_1f98c33a3fcd.slice/crio-60a6f6ea3a6f85b5356c1c37bb77e9a193ac4f6a8db76bf9fe1bb5ffa950c326 WatchSource:0}: Error finding container 60a6f6ea3a6f85b5356c1c37bb77e9a193ac4f6a8db76bf9fe1bb5ffa950c326: Status 404 returned error can't find the container with id 60a6f6ea3a6f85b5356c1c37bb77e9a193ac4f6a8db76bf9fe1bb5ffa950c326
Oct 31 13:45:45.919665 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:45.919593    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"]
Oct 31 13:45:46.819155 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:46.819056    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:46.873481 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:46.873434    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6" event={"ID":"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd","Type":"ContainerStarted","Data":"360603f0d465af1822bb2f1f9600feb44afe4c096cb0a17a88d64cb7676f8de0"}
Oct 31 13:45:46.873677 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:46.873505    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6" event={"ID":"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd","Type":"ContainerStarted","Data":"60a6f6ea3a6f85b5356c1c37bb77e9a193ac4f6a8db76bf9fe1bb5ffa950c326"}
Oct 31 13:45:46.894611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:46.894550    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6" podStartSLOduration=1.8945303249999998 podStartE2EDuration="1.894530325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:46.89246819 +0000 UTC m=+2544.290816837" watchObservedRunningTime="2026-10-31 13:45:46.894530325 +0000 UTC m=+2544.292878951"
Oct 31 13:45:47.505996 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:47.505947    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg" podUID="e5d14076-4049-4177-9a99-738b8d8e808b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:45:47.626896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:47.623732    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:47.883168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:47.883083    4428 generic.go:356] "Generic (PLEG): container finished" podID="8dd770fd-6968-41a0-ab2d-1f98c33a3fcd" containerID="360603f0d465af1822bb2f1f9600feb44afe4c096cb0a17a88d64cb7676f8de0" exitCode=0
Oct 31 13:45:47.883536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:47.883169    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6" event={"ID":"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd","Type":"ContainerDied","Data":"360603f0d465af1822bb2f1f9600feb44afe4c096cb0a17a88d64cb7676f8de0"}
Oct 31 13:45:48.275893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:48.273780    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:48.588415 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:48.588302    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:48.765254 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:48.765167    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:45:48.776460 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:48.776420    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n" podUID="1769894b-3274-4280-b0e2-a7056d77a96a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:45:49.071029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:49.070994    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"
Oct 31 13:45:49.141407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:49.141370    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-maas-api-xks-ca\") pod \"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd\" (UID: \"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd\") "
Oct 31 13:45:49.141619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:49.141531    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-maas-api-openshift-ca\") pod \"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd\" (UID: \"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd\") "
Oct 31 13:45:49.141619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:49.141583    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-kube-api-access-95prk\" (UniqueName: \"kubernetes.io/projected/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-kube-api-access-95prk\") pod \"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd\" (UID: \"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd\") "
Oct 31 13:45:49.142115 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:49.142076    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-maas-api-openshift-ca" pod "8dd770fd-6968-41a0-ab2d-1f98c33a3fcd" (UID: "8dd770fd-6968-41a0-ab2d-1f98c33a3fcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:49.142710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:49.142647    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:49.144393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:49.144367    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-kube-api-access-95prk" pod "8dd770fd-6968-41a0-ab2d-1f98c33a3fcd" (UID: "8dd770fd-6968-41a0-ab2d-1f98c33a3fcd"). InnerVolumeSpecName "kube-api-access-95prk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:49.144393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:49.144379    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-maas-api-xks-ca" pod "8dd770fd-6968-41a0-ab2d-1f98c33a3fcd" (UID: "8dd770fd-6968-41a0-ab2d-1f98c33a3fcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:45:49.244426 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:49.244391    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:49.244426 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:49.244424    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95prk\" (UniqueName: \"kubernetes.io/projected/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd-kube-api-access-95prk\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:49.668296 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:49.662590    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:49.901370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:49.901337    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"
Oct 31 13:45:49.901579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:49.901371    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6" event={"ID":"8dd770fd-6968-41a0-ab2d-1f98c33a3fcd","Type":"ContainerDied","Data":"60a6f6ea3a6f85b5356c1c37bb77e9a193ac4f6a8db76bf9fe1bb5ffa950c326"}
Oct 31 13:45:49.901579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:49.901419    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60a6f6ea3a6f85b5356c1c37bb77e9a193ac4f6a8db76bf9fe1bb5ffa950c326"
Oct 31 13:45:50.296597 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.296572    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:45:50.387590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.387549    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istiod-ca-cert\") pod \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") "
Oct 31 13:45:50.387983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.387680    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-workload-certs\") pod \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") "
Oct 31 13:45:50.387983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.387821    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-podinfo\") pod \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") "
Oct 31 13:45:50.387983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.387953    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-envoy\") pod \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") "
Oct 31 13:45:50.388234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.388024    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-workload-socket\") pod \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") "
Oct 31 13:45:50.388234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.388042    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-workload-certs" pod "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" (UID: "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:50.388234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.388057    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-data\") pod \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") "
Oct 31 13:45:50.388234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.388040    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istiod-ca-cert" pod "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" (UID: "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:50.388234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.388159    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-kube-api-access-fnnln\" (UniqueName: \"kubernetes.io/projected/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-kube-api-access-fnnln\") pod \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") "
Oct 31 13:45:50.388234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.388200    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-credential-socket\") pod \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") "
Oct 31 13:45:50.388561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.388271    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-token\" (UniqueName: \"kubernetes.io/projected/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-token\") pod \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\" (UID: \"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74\") "
Oct 31 13:45:50.388561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.388494    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-data" pod "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" (UID: "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:50.388912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.388883    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-credential-socket" pod "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" (UID: "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:50.389163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.389140    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-workload-socket" pod "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" (UID: "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:50.389311 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.389285    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.389402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.389316    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.389402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.389332    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.389402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.389345    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.391546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.390922    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-envoy" pod "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" (UID: "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:45:50.391546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.391328    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-podinfo" pod "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" (UID: "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:45:50.391546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.391333    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-kube-api-access-fnnln" pod "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" (UID: "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74"). InnerVolumeSpecName "kube-api-access-fnnln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:50.391759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.391667    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-token" pod "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" (UID: "9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:50.491231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.491189    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.491231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.491230    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.491450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.491247    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnnln\" (UniqueName: \"kubernetes.io/projected/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-kube-api-access-fnnln\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.491450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.491263    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.491450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.491278    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:50.572348 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.572252    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:50.658689 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:50.658645    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice/crio-f4ff7173cf463fa77294cb5afd3e25bd0fbc5e211a501b20b208a215eb151f31\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680bd885_b3c7_42f5_a3a8_4f2950d0b462.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:45:50.866243 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.866163    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:45:50.918976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.918938    4428 generic.go:356] "Generic (PLEG): container finished" podID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerID="04041c2d8e8caaea1b2dfbe3c3d2578a27ca932badca2417c101e56c6cfbea02" exitCode=0
Oct 31 13:45:50.919194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.919031    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"
Oct 31 13:45:50.919194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.919029    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" event={"ID":"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74","Type":"ContainerDied","Data":"04041c2d8e8caaea1b2dfbe3c3d2578a27ca932badca2417c101e56c6cfbea02"}
Oct 31 13:45:50.919194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.919103    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7" event={"ID":"9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74","Type":"ContainerDied","Data":"b7ded35ba08684ca3042a5d50715c15f4618c214e371d5028d2e1969979f8b18"}
Oct 31 13:45:50.919194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.919137    4428 scope.go:122] "RemoveContainer" containerID="04041c2d8e8caaea1b2dfbe3c3d2578a27ca932badca2417c101e56c6cfbea02"
Oct 31 13:45:50.935555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.935527    4428 scope.go:122] "RemoveContainer" containerID="571c05265fb8792d5e55bf6d8a08b690d0efb1dfa84fafa7319e6ce33250ec7a"
Oct 31 13:45:50.947381 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.947319    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"]
Oct 31 13:45:50.953262 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.953222    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fb50fa-openshift-default-5847d9656c-f7dt7"]
Oct 31 13:45:50.954240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.954221    4428 scope.go:122] "RemoveContainer" containerID="04041c2d8e8caaea1b2dfbe3c3d2578a27ca932badca2417c101e56c6cfbea02"
Oct 31 13:45:50.954711 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:50.954683    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04041c2d8e8caaea1b2dfbe3c3d2578a27ca932badca2417c101e56c6cfbea02\": container with ID starting with 04041c2d8e8caaea1b2dfbe3c3d2578a27ca932badca2417c101e56c6cfbea02 not found: ID does not exist" containerID="04041c2d8e8caaea1b2dfbe3c3d2578a27ca932badca2417c101e56c6cfbea02"
Oct 31 13:45:50.954782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.954723    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04041c2d8e8caaea1b2dfbe3c3d2578a27ca932badca2417c101e56c6cfbea02"} err="failed to get container status \"04041c2d8e8caaea1b2dfbe3c3d2578a27ca932badca2417c101e56c6cfbea02\": rpc error: code = NotFound desc = could not find container \"04041c2d8e8caaea1b2dfbe3c3d2578a27ca932badca2417c101e56c6cfbea02\": container with ID starting with 04041c2d8e8caaea1b2dfbe3c3d2578a27ca932badca2417c101e56c6cfbea02 not found: ID does not exist"
Oct 31 13:45:50.954782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.954748    4428 scope.go:122] "RemoveContainer" containerID="571c05265fb8792d5e55bf6d8a08b690d0efb1dfa84fafa7319e6ce33250ec7a"
Oct 31 13:45:50.955116 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:50.955087    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571c05265fb8792d5e55bf6d8a08b690d0efb1dfa84fafa7319e6ce33250ec7a\": container with ID starting with 571c05265fb8792d5e55bf6d8a08b690d0efb1dfa84fafa7319e6ce33250ec7a not found: ID does not exist" containerID="571c05265fb8792d5e55bf6d8a08b690d0efb1dfa84fafa7319e6ce33250ec7a"
Oct 31 13:45:50.955220 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:50.955115    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571c05265fb8792d5e55bf6d8a08b690d0efb1dfa84fafa7319e6ce33250ec7a"} err="failed to get container status \"571c05265fb8792d5e55bf6d8a08b690d0efb1dfa84fafa7319e6ce33250ec7a\": rpc error: code = NotFound desc = could not find container \"571c05265fb8792d5e55bf6d8a08b690d0efb1dfa84fafa7319e6ce33250ec7a\": container with ID starting with 571c05265fb8792d5e55bf6d8a08b690d0efb1dfa84fafa7319e6ce33250ec7a not found: ID does not exist"
Oct 31 13:45:51.413211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.413164    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:51.730894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.718706    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:51.744895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.741004    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"]
Oct 31 13:45:51.744895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.742429    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy"
Oct 31 13:45:51.744895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.742452    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy"
Oct 31 13:45:51.744895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.742539    4428 cpu_manager.go:414] "containerMap: removing container" podUID="8dd770fd-6968-41a0-ab2d-1f98c33a3fcd" containerName="revoke-keys"
Oct 31 13:45:51.744895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.742549    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd770fd-6968-41a0-ab2d-1f98c33a3fcd" containerName="revoke-keys"
Oct 31 13:45:51.744895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.742743    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy"
Oct 31 13:45:51.744895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.742762    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd770fd-6968-41a0-ab2d-1f98c33a3fcd" containerName="revoke-keys"
Oct 31 13:45:51.764673 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.764243    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"]
Oct 31 13:45:51.764673 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.764412    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"
Oct 31 13:45:51.773159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.773133    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:45:51.774263 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.773967    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 13:45:51.857344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.857305    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4021c4a5-c6b5-44fa-be97-26e412f3af20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm\" (UID: \"4021c4a5-c6b5-44fa-be97-26e412f3af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"
Oct 31 13:45:51.857545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.857463    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4021c4a5-c6b5-44fa-be97-26e412f3af20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm\" (UID: \"4021c4a5-c6b5-44fa-be97-26e412f3af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"
Oct 31 13:45:51.857545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.857538    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sgx6\" (UniqueName: \"kubernetes.io/projected/4021c4a5-c6b5-44fa-be97-26e412f3af20-kube-api-access-2sgx6\") pod \"maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm\" (UID: \"4021c4a5-c6b5-44fa-be97-26e412f3af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"
Oct 31 13:45:51.959714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.959672    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4021c4a5-c6b5-44fa-be97-26e412f3af20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm\" (UID: \"4021c4a5-c6b5-44fa-be97-26e412f3af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"
Oct 31 13:45:51.959923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.959785    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4021c4a5-c6b5-44fa-be97-26e412f3af20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm\" (UID: \"4021c4a5-c6b5-44fa-be97-26e412f3af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"
Oct 31 13:45:51.959923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.959839    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sgx6\" (UniqueName: \"kubernetes.io/projected/4021c4a5-c6b5-44fa-be97-26e412f3af20-kube-api-access-2sgx6\") pod \"maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm\" (UID: \"4021c4a5-c6b5-44fa-be97-26e412f3af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"
Oct 31 13:45:51.960682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.960660    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4021c4a5-c6b5-44fa-be97-26e412f3af20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm\" (UID: \"4021c4a5-c6b5-44fa-be97-26e412f3af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"
Oct 31 13:45:51.963187 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.963159    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4021c4a5-c6b5-44fa-be97-26e412f3af20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm\" (UID: \"4021c4a5-c6b5-44fa-be97-26e412f3af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"
Oct 31 13:45:51.976428 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:51.974605    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sgx6\" (UniqueName: \"kubernetes.io/projected/4021c4a5-c6b5-44fa-be97-26e412f3af20-kube-api-access-2sgx6\") pod \"maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm\" (UID: \"4021c4a5-c6b5-44fa-be97-26e412f3af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"
Oct 31 13:45:52.085005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:52.084920    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"
Oct 31 13:45:52.387827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:52.387796    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"]
Oct 31 13:45:52.388362 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:45:52.388337    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4021c4a5_c6b5_44fa_be97_26e412f3af20.slice/crio-368f63d035b8a83b8ea10f5893600caa11a4ef6de90f4e2c0462f6404476aa42 WatchSource:0}: Error finding container 368f63d035b8a83b8ea10f5893600caa11a4ef6de90f4e2c0462f6404476aa42: Status 404 returned error can't find the container with id 368f63d035b8a83b8ea10f5893600caa11a4ef6de90f4e2c0462f6404476aa42
Oct 31 13:45:52.737008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:52.736963    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:52.760001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:52.759971    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" path="/var/lib/kubelet/pods/9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74/volumes"
Oct 31 13:45:52.944244 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:52.944210    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm" event={"ID":"4021c4a5-c6b5-44fa-be97-26e412f3af20","Type":"ContainerStarted","Data":"368f63d035b8a83b8ea10f5893600caa11a4ef6de90f4e2c0462f6404476aa42"}
Oct 31 13:45:53.725401 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:53.723014    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:53.955740 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:53.955699    4428 generic.go:356] "Generic (PLEG): container finished" podID="4021c4a5-c6b5-44fa-be97-26e412f3af20" containerID="2e3d33c968d056f4bcc1588cca3df926b9483583d0884a73cc32588f73026028" exitCode=0
Oct 31 13:45:53.956278 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:53.955754    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm" event={"ID":"4021c4a5-c6b5-44fa-be97-26e412f3af20","Type":"ContainerDied","Data":"2e3d33c968d056f4bcc1588cca3df926b9483583d0884a73cc32588f73026028"}
Oct 31 13:45:54.540720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:54.540681    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:54.752895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:54.744154    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:55.189911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:55.189883    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"
Oct 31 13:45:55.337113 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:55.337021    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4021c4a5-c6b5-44fa-be97-26e412f3af20-kube-api-access-2sgx6\" (UniqueName: \"kubernetes.io/projected/4021c4a5-c6b5-44fa-be97-26e412f3af20-kube-api-access-2sgx6\") pod \"4021c4a5-c6b5-44fa-be97-26e412f3af20\" (UID: \"4021c4a5-c6b5-44fa-be97-26e412f3af20\") "
Oct 31 13:45:55.337306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:55.337207    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4021c4a5-c6b5-44fa-be97-26e412f3af20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4021c4a5-c6b5-44fa-be97-26e412f3af20-maas-api-openshift-ca\") pod \"4021c4a5-c6b5-44fa-be97-26e412f3af20\" (UID: \"4021c4a5-c6b5-44fa-be97-26e412f3af20\") "
Oct 31 13:45:55.337389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:55.337320    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4021c4a5-c6b5-44fa-be97-26e412f3af20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4021c4a5-c6b5-44fa-be97-26e412f3af20-maas-api-xks-ca\") pod \"4021c4a5-c6b5-44fa-be97-26e412f3af20\" (UID: \"4021c4a5-c6b5-44fa-be97-26e412f3af20\") "
Oct 31 13:45:55.337663 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:55.337633    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4021c4a5-c6b5-44fa-be97-26e412f3af20-maas-api-openshift-ca" pod "4021c4a5-c6b5-44fa-be97-26e412f3af20" (UID: "4021c4a5-c6b5-44fa-be97-26e412f3af20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:55.338148 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:55.338121    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4021c4a5-c6b5-44fa-be97-26e412f3af20-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:55.339533 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:55.339509    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4021c4a5-c6b5-44fa-be97-26e412f3af20-maas-api-xks-ca" pod "4021c4a5-c6b5-44fa-be97-26e412f3af20" (UID: "4021c4a5-c6b5-44fa-be97-26e412f3af20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:45:55.339618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:55.339553    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4021c4a5-c6b5-44fa-be97-26e412f3af20-kube-api-access-2sgx6" pod "4021c4a5-c6b5-44fa-be97-26e412f3af20" (UID: "4021c4a5-c6b5-44fa-be97-26e412f3af20"). InnerVolumeSpecName "kube-api-access-2sgx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:55.384501 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:55.384455    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice/crio-f4ff7173cf463fa77294cb5afd3e25bd0fbc5e211a501b20b208a215eb151f31\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680bd885_b3c7_42f5_a3a8_4f2950d0b462.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:45:55.439774 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:55.439737    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sgx6\" (UniqueName: \"kubernetes.io/projected/4021c4a5-c6b5-44fa-be97-26e412f3af20-kube-api-access-2sgx6\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:55.439774 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:55.439778    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4021c4a5-c6b5-44fa-be97-26e412f3af20-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:55.574131 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:55.574097    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:55.758525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:55.758435    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:55.939748 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:55.938388    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:56.025661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.025556    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm" event={"ID":"4021c4a5-c6b5-44fa-be97-26e412f3af20","Type":"ContainerDied","Data":"368f63d035b8a83b8ea10f5893600caa11a4ef6de90f4e2c0462f6404476aa42"}
Oct 31 13:45:56.035952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.035910    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="368f63d035b8a83b8ea10f5893600caa11a4ef6de90f4e2c0462f6404476aa42"
Oct 31 13:45:56.036271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.025976    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"
Oct 31 13:45:56.713980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.690658    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:56.798160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.798130    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"
Oct 31 13:45:56.802320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.802293    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"
Oct 31 13:45:56.886010 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.885904    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5d14076-4049-4177-9a99-738b8d8e808b-kube-api-access-rc2xf\" (UniqueName: \"kubernetes.io/projected/e5d14076-4049-4177-9a99-738b8d8e808b-kube-api-access-rc2xf\") pod \"e5d14076-4049-4177-9a99-738b8d8e808b\" (UID: \"e5d14076-4049-4177-9a99-738b8d8e808b\") "
Oct 31 13:45:56.886207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.886070    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1769894b-3274-4280-b0e2-a7056d77a96a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1769894b-3274-4280-b0e2-a7056d77a96a-plugins-config-volume\") pod \"1769894b-3274-4280-b0e2-a7056d77a96a\" (UID: \"1769894b-3274-4280-b0e2-a7056d77a96a\") "
Oct 31 13:45:56.886207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.886151    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5d14076-4049-4177-9a99-738b8d8e808b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d14076-4049-4177-9a99-738b8d8e808b-plugins-config-volume\") pod \"e5d14076-4049-4177-9a99-738b8d8e808b\" (UID: \"e5d14076-4049-4177-9a99-738b8d8e808b\") "
Oct 31 13:45:56.886207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.886182    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1769894b-3274-4280-b0e2-a7056d77a96a-kube-api-access-rxmvl\" (UniqueName: \"kubernetes.io/projected/1769894b-3274-4280-b0e2-a7056d77a96a-kube-api-access-rxmvl\") pod \"1769894b-3274-4280-b0e2-a7056d77a96a\" (UID: \"1769894b-3274-4280-b0e2-a7056d77a96a\") "
Oct 31 13:45:56.886557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.886526    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1769894b-3274-4280-b0e2-a7056d77a96a-plugins-config-volume" pod "1769894b-3274-4280-b0e2-a7056d77a96a" (UID: "1769894b-3274-4280-b0e2-a7056d77a96a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:56.886739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.886717    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d14076-4049-4177-9a99-738b8d8e808b-plugins-config-volume" pod "e5d14076-4049-4177-9a99-738b8d8e808b" (UID: "e5d14076-4049-4177-9a99-738b8d8e808b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:45:56.889408 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.887165    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1769894b-3274-4280-b0e2-a7056d77a96a-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:56.889408 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.887201    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d14076-4049-4177-9a99-738b8d8e808b-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:56.889644 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.889620    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1769894b-3274-4280-b0e2-a7056d77a96a-kube-api-access-rxmvl" pod "1769894b-3274-4280-b0e2-a7056d77a96a" (UID: "1769894b-3274-4280-b0e2-a7056d77a96a"). InnerVolumeSpecName "kube-api-access-rxmvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:56.890571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.890544    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d14076-4049-4177-9a99-738b8d8e808b-kube-api-access-rc2xf" pod "e5d14076-4049-4177-9a99-738b8d8e808b" (UID: "e5d14076-4049-4177-9a99-738b8d8e808b"). InnerVolumeSpecName "kube-api-access-rc2xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:45:56.988437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.988397    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc2xf\" (UniqueName: \"kubernetes.io/projected/e5d14076-4049-4177-9a99-738b8d8e808b-kube-api-access-rc2xf\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:56.988437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:56.988431    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxmvl\" (UniqueName: \"kubernetes.io/projected/1769894b-3274-4280-b0e2-a7056d77a96a-kube-api-access-rxmvl\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:45:57.034674 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.034640    4428 generic.go:356] "Generic (PLEG): container finished" podID="1769894b-3274-4280-b0e2-a7056d77a96a" containerID="d4fe255e47338f623d48d69d838488b47d653bb8499600f7cc8047a29e178032" exitCode=0
Oct 31 13:45:57.034912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.034728    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"
Oct 31 13:45:57.034912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.034730    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n" event={"ID":"1769894b-3274-4280-b0e2-a7056d77a96a","Type":"ContainerDied","Data":"d4fe255e47338f623d48d69d838488b47d653bb8499600f7cc8047a29e178032"}
Oct 31 13:45:57.034912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.034839    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n" event={"ID":"1769894b-3274-4280-b0e2-a7056d77a96a","Type":"ContainerDied","Data":"cc856ef7b639192c95bd0fab22d95f43029e2e3d5afdc8f1d33e8b7742d6e6ca"}
Oct 31 13:45:57.034912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.034905    4428 scope.go:122] "RemoveContainer" containerID="d4fe255e47338f623d48d69d838488b47d653bb8499600f7cc8047a29e178032"
Oct 31 13:45:57.036634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.036612    4428 generic.go:356] "Generic (PLEG): container finished" podID="e5d14076-4049-4177-9a99-738b8d8e808b" containerID="dfbd850fd86091acd667d73c628781f3e3de84147b5fe30ebdf4e6a57c239648" exitCode=0
Oct 31 13:45:57.036757 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.036643    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg" event={"ID":"e5d14076-4049-4177-9a99-738b8d8e808b","Type":"ContainerDied","Data":"dfbd850fd86091acd667d73c628781f3e3de84147b5fe30ebdf4e6a57c239648"}
Oct 31 13:45:57.036757 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.036679    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"
Oct 31 13:45:57.036757 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.036709    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg" event={"ID":"e5d14076-4049-4177-9a99-738b8d8e808b","Type":"ContainerDied","Data":"73e8947b6f88569bfa547c317bf02d3e4b31c2c7d31de5b1e75d6213878d632a"}
Oct 31 13:45:57.048673 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.048655    4428 scope.go:122] "RemoveContainer" containerID="d4fe255e47338f623d48d69d838488b47d653bb8499600f7cc8047a29e178032"
Oct 31 13:45:57.049021 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:57.048993    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4fe255e47338f623d48d69d838488b47d653bb8499600f7cc8047a29e178032\": container with ID starting with d4fe255e47338f623d48d69d838488b47d653bb8499600f7cc8047a29e178032 not found: ID does not exist" containerID="d4fe255e47338f623d48d69d838488b47d653bb8499600f7cc8047a29e178032"
Oct 31 13:45:57.049097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.049033    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4fe255e47338f623d48d69d838488b47d653bb8499600f7cc8047a29e178032"} err="failed to get container status \"d4fe255e47338f623d48d69d838488b47d653bb8499600f7cc8047a29e178032\": rpc error: code = NotFound desc = could not find container \"d4fe255e47338f623d48d69d838488b47d653bb8499600f7cc8047a29e178032\": container with ID starting with d4fe255e47338f623d48d69d838488b47d653bb8499600f7cc8047a29e178032 not found: ID does not exist"
Oct 31 13:45:57.049097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.049059    4428 scope.go:122] "RemoveContainer" containerID="dfbd850fd86091acd667d73c628781f3e3de84147b5fe30ebdf4e6a57c239648"
Oct 31 13:45:57.060322 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.060304    4428 scope.go:122] "RemoveContainer" containerID="dfbd850fd86091acd667d73c628781f3e3de84147b5fe30ebdf4e6a57c239648"
Oct 31 13:45:57.060623 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:45:57.060605    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfbd850fd86091acd667d73c628781f3e3de84147b5fe30ebdf4e6a57c239648\": container with ID starting with dfbd850fd86091acd667d73c628781f3e3de84147b5fe30ebdf4e6a57c239648 not found: ID does not exist" containerID="dfbd850fd86091acd667d73c628781f3e3de84147b5fe30ebdf4e6a57c239648"
Oct 31 13:45:57.060680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.060632    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfbd850fd86091acd667d73c628781f3e3de84147b5fe30ebdf4e6a57c239648"} err="failed to get container status \"dfbd850fd86091acd667d73c628781f3e3de84147b5fe30ebdf4e6a57c239648\": rpc error: code = NotFound desc = could not find container \"dfbd850fd86091acd667d73c628781f3e3de84147b5fe30ebdf4e6a57c239648\": container with ID starting with dfbd850fd86091acd667d73c628781f3e3de84147b5fe30ebdf4e6a57c239648 not found: ID does not exist"
Oct 31 13:45:57.068606 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.068581    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"]
Oct 31 13:45:57.077163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.077139    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fb50fa-c8f6d95f9-sn2vg"]
Oct 31 13:45:57.088452 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.088424    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"]
Oct 31 13:45:57.092923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.092901    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fb50fa-5cdd7d8ddf-cnw9n"]
Oct 31 13:45:57.515894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:57.514504    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:58.588638 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:58.588598    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"]
Oct 31 13:45:58.659890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:58.659835    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:45:58.661766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:58.661731    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"]
Oct 31 13:45:58.759521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:58.759485    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1769894b-3274-4280-b0e2-a7056d77a96a" path="/var/lib/kubelet/pods/1769894b-3274-4280-b0e2-a7056d77a96a/volumes"
Oct 31 13:45:58.759990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:58.759972    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5d14076-4049-4177-9a99-738b8d8e808b" path="/var/lib/kubelet/pods/e5d14076-4049-4177-9a99-738b8d8e808b/volumes"
Oct 31 13:45:58.847416 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:45:58.847318    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:00.063905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:00.060805    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:01.128892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:01.118107    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:46:01.235891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:01.232999    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:46:01.424016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:01.418493    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf" podUID="afe8d194-92d6-471f-8233-15ba5c15b33a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.218:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:46:01.520891 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:01.520686    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680bd885_b3c7_42f5_a3a8_4f2950d0b462.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice/crio-f4ff7173cf463fa77294cb5afd3e25bd0fbc5e211a501b20b208a215eb151f31\": RecentStats: unable to find data in memory cache]"
Oct 31 13:46:01.799134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:01.799088    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"]
Oct 31 13:46:01.814788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:01.814748    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"]
Oct 31 13:46:01.949758 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:01.949714    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:02.115784 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:02.115688    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:03.138981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:03.138940    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:03.589815 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:03.589771    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8" podUID="67b60c41-c1fe-4112-88ea-423d2dc01997" containerName="payload-processing" containerID="cri-o://e61c40fec420930433b207f0f2357e153b08c6cf356d9d9e2e51a2eaa4d019bb" gracePeriod=25
Oct 31 13:46:03.662687 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:03.662634    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj" podUID="1fb736b6-9425-445b-85bb-be3f5a9598d1" containerName="payload-pre-processing" containerID="cri-o://b4e65dadf04e7eb0b0103a88896d02ebdd5495eaec350a62865542013c8f9fa2" gracePeriod=25
Oct 31 13:46:03.886961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:03.886925    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"
Oct 31 13:46:03.967785 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:03.967694    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b60c41-c1fe-4112-88ea-423d2dc01997-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b60c41-c1fe-4112-88ea-423d2dc01997-plugins-config-volume\") pod \"67b60c41-c1fe-4112-88ea-423d2dc01997\" (UID: \"67b60c41-c1fe-4112-88ea-423d2dc01997\") "
Oct 31 13:46:03.968032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:03.967984    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b60c41-c1fe-4112-88ea-423d2dc01997-kube-api-access-c659c\" (UniqueName: \"kubernetes.io/projected/67b60c41-c1fe-4112-88ea-423d2dc01997-kube-api-access-c659c\") pod \"67b60c41-c1fe-4112-88ea-423d2dc01997\" (UID: \"67b60c41-c1fe-4112-88ea-423d2dc01997\") "
Oct 31 13:46:03.968256 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:03.968226    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b60c41-c1fe-4112-88ea-423d2dc01997-plugins-config-volume" pod "67b60c41-c1fe-4112-88ea-423d2dc01997" (UID: "67b60c41-c1fe-4112-88ea-423d2dc01997"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:03.968777 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:03.968753    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b60c41-c1fe-4112-88ea-423d2dc01997-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:03.970444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:03.970419    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b60c41-c1fe-4112-88ea-423d2dc01997-kube-api-access-c659c" pod "67b60c41-c1fe-4112-88ea-423d2dc01997" (UID: "67b60c41-c1fe-4112-88ea-423d2dc01997"). InnerVolumeSpecName "kube-api-access-c659c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:04.079921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:04.079605    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:04.083267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:04.082799    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c659c\" (UniqueName: \"kubernetes.io/projected/67b60c41-c1fe-4112-88ea-423d2dc01997-kube-api-access-c659c\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:04.397259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:04.397220    4428 generic.go:356] "Generic (PLEG): container finished" podID="67b60c41-c1fe-4112-88ea-423d2dc01997" containerID="e61c40fec420930433b207f0f2357e153b08c6cf356d9d9e2e51a2eaa4d019bb" exitCode=0
Oct 31 13:46:04.397722 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:04.397270    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"
Oct 31 13:46:04.397722 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:04.397301    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8" event={"ID":"67b60c41-c1fe-4112-88ea-423d2dc01997","Type":"ContainerDied","Data":"e61c40fec420930433b207f0f2357e153b08c6cf356d9d9e2e51a2eaa4d019bb"}
Oct 31 13:46:04.397722 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:04.397362    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8" event={"ID":"67b60c41-c1fe-4112-88ea-423d2dc01997","Type":"ContainerDied","Data":"f75339b7319e982149b2674442115c02906934b42e2e116c41c259db3360e389"}
Oct 31 13:46:04.397722 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:04.397384    4428 scope.go:122] "RemoveContainer" containerID="e61c40fec420930433b207f0f2357e153b08c6cf356d9d9e2e51a2eaa4d019bb"
Oct 31 13:46:04.409258 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:04.409241    4428 scope.go:122] "RemoveContainer" containerID="e61c40fec420930433b207f0f2357e153b08c6cf356d9d9e2e51a2eaa4d019bb"
Oct 31 13:46:04.409560 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:04.409536    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e61c40fec420930433b207f0f2357e153b08c6cf356d9d9e2e51a2eaa4d019bb\": container with ID starting with e61c40fec420930433b207f0f2357e153b08c6cf356d9d9e2e51a2eaa4d019bb not found: ID does not exist" containerID="e61c40fec420930433b207f0f2357e153b08c6cf356d9d9e2e51a2eaa4d019bb"
Oct 31 13:46:04.409635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:04.409568    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e61c40fec420930433b207f0f2357e153b08c6cf356d9d9e2e51a2eaa4d019bb"} err="failed to get container status \"e61c40fec420930433b207f0f2357e153b08c6cf356d9d9e2e51a2eaa4d019bb\": rpc error: code = NotFound desc = could not find container \"e61c40fec420930433b207f0f2357e153b08c6cf356d9d9e2e51a2eaa4d019bb\": container with ID starting with e61c40fec420930433b207f0f2357e153b08c6cf356d9d9e2e51a2eaa4d019bb not found: ID does not exist"
Oct 31 13:46:04.425561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:04.425535    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"]
Oct 31 13:46:04.430149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:04.430123    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4232c-5f5c7fd745-g7pp8"]
Oct 31 13:46:04.760506 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:04.760465    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b60c41-c1fe-4112-88ea-423d2dc01997" path="/var/lib/kubelet/pods/67b60c41-c1fe-4112-88ea-423d2dc01997/volumes"
Oct 31 13:46:04.945113 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:04.945076    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"]
Oct 31 13:46:04.962675 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:04.962639    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"]
Oct 31 13:46:05.097819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:05.097726    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:05.257543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:05.257503    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:05.922678 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:05.922643    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:06.127478 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:06.123031    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:06.307943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:06.303694    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:06.411075 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:06.411040    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"]
Oct 31 13:46:06.424915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:06.424817    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62778a-g6rgm"]
Oct 31 13:46:06.541001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:06.540953    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:06.759267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:06.759235    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4021c4a5-c6b5-44fa-be97-26e412f3af20" path="/var/lib/kubelet/pods/4021c4a5-c6b5-44fa-be97-26e412f3af20/volumes"
Oct 31 13:46:06.799587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:06.799534    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f" podUID="9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c" containerName="payload-processing" containerID="cri-o://9a13e70784f0e73ac0cf3c3617cc9881af581ba1afca128a128580378b8d54fe" gracePeriod=25
Oct 31 13:46:06.809697 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:06.809668    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:06.815741 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:06.815711    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7" podUID="8c73f7d7-3bac-40e8-b386-3742d5a73225" containerName="payload-pre-processing" containerID="cri-o://d70b97a5f61954d3edfbd4a0196f8032ee138f06a4290d8b8793fcc42836bddb" gracePeriod=25
Oct 31 13:46:07.108067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:07.108042    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"
Oct 31 13:46:07.151400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:07.151358    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c-kube-api-access-hrnlq\" (UniqueName: \"kubernetes.io/projected/9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c-kube-api-access-hrnlq\") pod \"9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c\" (UID: \"9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c\") "
Oct 31 13:46:07.151847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:07.151818    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c-plugins-config-volume\") pod \"9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c\" (UID: \"9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c\") "
Oct 31 13:46:07.152477 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:07.152453    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c-plugins-config-volume" pod "9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c" (UID: "9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:07.152655 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:07.152635    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:07.155079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:07.155048    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c-kube-api-access-hrnlq" pod "9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c" (UID: "9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c"). InnerVolumeSpecName "kube-api-access-hrnlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:07.254398 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:07.254356    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrnlq\" (UniqueName: \"kubernetes.io/projected/9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c-kube-api-access-hrnlq\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:07.457091 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:07.456994    4428 generic.go:356] "Generic (PLEG): container finished" podID="9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c" containerID="9a13e70784f0e73ac0cf3c3617cc9881af581ba1afca128a128580378b8d54fe" exitCode=0
Oct 31 13:46:07.457091 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:07.457041    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"
Oct 31 13:46:07.457363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:07.457095    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f" event={"ID":"9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c","Type":"ContainerDied","Data":"9a13e70784f0e73ac0cf3c3617cc9881af581ba1afca128a128580378b8d54fe"}
Oct 31 13:46:07.457363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:07.457154    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f" event={"ID":"9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c","Type":"ContainerDied","Data":"bc24aa524eaf6f5cba0adc11f548b7e7e50b137673153ae8cdff1c8a3afe9145"}
Oct 31 13:46:07.457363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:07.457183    4428 scope.go:122] "RemoveContainer" containerID="9a13e70784f0e73ac0cf3c3617cc9881af581ba1afca128a128580378b8d54fe"
Oct 31 13:46:07.469731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:07.469708    4428 scope.go:122] "RemoveContainer" containerID="9a13e70784f0e73ac0cf3c3617cc9881af581ba1afca128a128580378b8d54fe"
Oct 31 13:46:07.470094 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:07.470074    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a13e70784f0e73ac0cf3c3617cc9881af581ba1afca128a128580378b8d54fe\": container with ID starting with 9a13e70784f0e73ac0cf3c3617cc9881af581ba1afca128a128580378b8d54fe not found: ID does not exist" containerID="9a13e70784f0e73ac0cf3c3617cc9881af581ba1afca128a128580378b8d54fe"
Oct 31 13:46:07.470152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:07.470104    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a13e70784f0e73ac0cf3c3617cc9881af581ba1afca128a128580378b8d54fe"} err="failed to get container status \"9a13e70784f0e73ac0cf3c3617cc9881af581ba1afca128a128580378b8d54fe\": rpc error: code = NotFound desc = could not find container \"9a13e70784f0e73ac0cf3c3617cc9881af581ba1afca128a128580378b8d54fe\": container with ID starting with 9a13e70784f0e73ac0cf3c3617cc9881af581ba1afca128a128580378b8d54fe not found: ID does not exist"
Oct 31 13:46:07.494976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:07.494949    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"]
Oct 31 13:46:07.509101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:07.509077    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62778a-649fbc5df5-49w6f"]
Oct 31 13:46:07.942481 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:07.942442    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:08.781967 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:08.781917    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c" path="/var/lib/kubelet/pods/9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c/volumes"
Oct 31 13:46:08.799775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:08.799741    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:09.721640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:09.721586    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:09.945778 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:09.945736    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t" podUID="530cfbed-6986-4f00-9eb5-bb0d7985827e" containerName="payload-processing" containerID="cri-o://86bb0549b47fbe808d32b79a071312394d443405c1fa078cb82c5ade407dfc90" gracePeriod=25
Oct 31 13:46:09.963463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:09.963423    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl" podUID="5d8cf8de-bfa2-4e2d-8069-e1b1132fe426" containerName="payload-pre-processing" containerID="cri-o://612cee8557ca690e42db9e275a8b012bea92750513f5fd2ab4bb6675d53403b5" gracePeriod=25
Oct 31 13:46:10.465087 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:10.465049    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice/crio-f4ff7173cf463fa77294cb5afd3e25bd0fbc5e211a501b20b208a215eb151f31\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680bd885_b3c7_42f5_a3a8_4f2950d0b462.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:46:10.696342 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:10.696293    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:10.793721 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:10.793633    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7" podUID="8c73f7d7-3bac-40e8-b386-3742d5a73225" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:46:11.579824 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:11.579772    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680bd885_b3c7_42f5_a3a8_4f2950d0b462.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice/crio-f4ff7173cf463fa77294cb5afd3e25bd0fbc5e211a501b20b208a215eb151f31\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:46:11.699246 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:11.698388    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:12.253024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:12.252980    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj" podUID="1fb736b6-9425-445b-85bb-be3f5a9598d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:46:12.456608 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:12.456568    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:12.690267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:12.690175    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:12.875444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:12.875390    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:12.942207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:12.942119    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"]
Oct 31 13:46:12.951347 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:12.951317    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f295-xwzd6"]
Oct 31 13:46:13.116818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:13.116786    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:13.447316 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:13.447261    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:14.116896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:14.111497    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:14.316098 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:14.316060    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:14.682301 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:14.682264    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:14.757639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:14.757605    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dd770fd-6968-41a0-ab2d-1f98c33a3fcd" path="/var/lib/kubelet/pods/8dd770fd-6968-41a0-ab2d-1f98c33a3fcd/volumes"
Oct 31 13:46:14.848177 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:14.848141    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:14.974990 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:14.974955    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"]
Oct 31 13:46:14.975259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:14.975232    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy" containerID="cri-o://ec70f8a05fea67eac9a8733e166c147282ea548d10bffd295ea779a20c5bd8a8" gracePeriod=30
Oct 31 13:46:15.011475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:15.011437    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:15.190896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:15.190064    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:15.828662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:15.828623    4428 patch_prober.go:29] interesting pod/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:46:15.828662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:15.828678    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:46:16.179313 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:16.173221    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:16.680892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:16.680574    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:17.142285 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.142245    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:17.328770 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.328734    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:17.520064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.520029    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"]
Oct 31 13:46:17.521246 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521220    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy"
Oct 31 13:46:17.521246 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521245    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy"
Oct 31 13:46:17.521454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521259    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c" containerName="payload-processing"
Oct 31 13:46:17.521454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521267    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c" containerName="payload-processing"
Oct 31 13:46:17.521454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521286    4428 cpu_manager.go:414] "containerMap: removing container" podUID="1769894b-3274-4280-b0e2-a7056d77a96a" containerName="payload-processing"
Oct 31 13:46:17.521454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521293    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1769894b-3274-4280-b0e2-a7056d77a96a" containerName="payload-processing"
Oct 31 13:46:17.521454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521318    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e5d14076-4049-4177-9a99-738b8d8e808b" containerName="payload-pre-processing"
Oct 31 13:46:17.521454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521325    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5d14076-4049-4177-9a99-738b8d8e808b" containerName="payload-pre-processing"
Oct 31 13:46:17.521454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521365    4428 cpu_manager.go:414] "containerMap: removing container" podUID="67b60c41-c1fe-4112-88ea-423d2dc01997" containerName="payload-processing"
Oct 31 13:46:17.521454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521372    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b60c41-c1fe-4112-88ea-423d2dc01997" containerName="payload-processing"
Oct 31 13:46:17.521454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521383    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4021c4a5-c6b5-44fa-be97-26e412f3af20" containerName="revoke-keys"
Oct 31 13:46:17.521454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521389    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4021c4a5-c6b5-44fa-be97-26e412f3af20" containerName="revoke-keys"
Oct 31 13:46:17.522003 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521585    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb5ac9f-dc1c-4c44-95c5-9539ce62c50c" containerName="payload-processing"
Oct 31 13:46:17.522003 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521596    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1769894b-3274-4280-b0e2-a7056d77a96a" containerName="payload-processing"
Oct 31 13:46:17.522003 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521626    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5d14076-4049-4177-9a99-738b8d8e808b" containerName="payload-pre-processing"
Oct 31 13:46:17.522003 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521639    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc8b218-3e92-4c29-b42f-9eb3dbdd2d74" containerName="istio-proxy"
Oct 31 13:46:17.522003 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521650    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b60c41-c1fe-4112-88ea-423d2dc01997" containerName="payload-processing"
Oct 31 13:46:17.522003 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.521660    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4021c4a5-c6b5-44fa-be97-26e412f3af20" containerName="revoke-keys"
Oct 31 13:46:17.528026 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.527992    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.535311 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.535286    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-612b46-openshift-default-dockercfg-7m976\""
Oct 31 13:46:17.539292 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.539265    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"]
Oct 31 13:46:17.657502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.656319    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-credential-socket\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.657502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.656425    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-token\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.657502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.656464    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-podinfo\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.657502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.656511    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-workload-socket\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.657502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.656644    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-envoy\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.657502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.656764    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-data\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.657502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.656825    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istiod-ca-cert\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.657502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.656904    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gtms\" (UniqueName: \"kubernetes.io/projected/fdbe827a-0f8e-41f0-a10e-0893d404efbe-kube-api-access-4gtms\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.657502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.656975    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-workload-certs\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.658221 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.657742    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:17.763944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.763912    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-envoy\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.765361 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.765332    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-data\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.766009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.765989    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istiod-ca-cert\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.773503 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.765900    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-data\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.773909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.773884    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istiod-ca-cert\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.774474 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.774081    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gtms\" (UniqueName: \"kubernetes.io/projected/fdbe827a-0f8e-41f0-a10e-0893d404efbe-kube-api-access-4gtms\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.791893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.774665    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-workload-certs\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.800244 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.792544    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-workload-certs\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.801070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.800100    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-envoy\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.801262 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.800595    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-credential-socket\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.801495 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.801474    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-token\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.809439 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.800993    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-credential-socket\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.809836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.809814    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-podinfo\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.824982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.824849    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-workload-socket\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.826040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.826014    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-workload-socket\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.836839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.836805    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:17.852479 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.851965    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gtms\" (UniqueName: \"kubernetes.io/projected/fdbe827a-0f8e-41f0-a10e-0893d404efbe-kube-api-access-4gtms\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.853812 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.853786    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-podinfo\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.861959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.861699    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-token\") pod \"e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") " pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:17.862518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:17.862298    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:18.246039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:18.246006    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"]
Oct 31 13:46:18.246626 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:46:18.246592    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbe827a_0f8e_41f0_a10e_0893d404efbe.slice/crio-fa705bbd3889f36066eba8aec4290966045e15f05d037f8599682234e1ebfdba WatchSource:0}: Error finding container fa705bbd3889f36066eba8aec4290966045e15f05d037f8599682234e1ebfdba: Status 404 returned error can't find the container with id fa705bbd3889f36066eba8aec4290966045e15f05d037f8599682234e1ebfdba
Oct 31 13:46:18.249769 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:18.249728    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:46:18.249890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:18.249829    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:46:18.249890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:18.249882    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:46:18.462943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:18.450137    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:18.586323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:18.586214    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb" event={"ID":"fdbe827a-0f8e-41f0-a10e-0893d404efbe","Type":"ContainerStarted","Data":"e6ee5852d7a7f2ef0706c0175847e8eb8744833445e552a8018fd670b0eec565"}
Oct 31 13:46:18.586323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:18.586289    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb" event={"ID":"fdbe827a-0f8e-41f0-a10e-0893d404efbe","Type":"ContainerStarted","Data":"fa705bbd3889f36066eba8aec4290966045e15f05d037f8599682234e1ebfdba"}
Oct 31 13:46:18.622900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:18.622819    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb" podStartSLOduration=1.622802945 podStartE2EDuration="1.622802945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:46:18.61810188 +0000 UTC m=+2576.016450506" watchObservedRunningTime="2026-10-31 13:46:18.622802945 +0000 UTC m=+2576.021151569"
Oct 31 13:46:18.818933 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:18.807414    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:46:18.876185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:18.876040    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:19.392934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:19.392896    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:19.418348 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:19.418306    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl" podUID="5d8cf8de-bfa2-4e2d-8069-e1b1132fe426" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:46:19.418554 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:19.418320    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t" podUID="530cfbed-6986-4f00-9eb5-bb0d7985827e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:46:19.684487 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:19.684400    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice/crio-f4ff7173cf463fa77294cb5afd3e25bd0fbc5e211a501b20b208a215eb151f31\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680bd885_b3c7_42f5_a3a8_4f2950d0b462.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:46:19.684653 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:19.684488    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680bd885_b3c7_42f5_a3a8_4f2950d0b462.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice/crio-f4ff7173cf463fa77294cb5afd3e25bd0fbc5e211a501b20b208a215eb151f31\": RecentStats: unable to find data in memory cache]"
Oct 31 13:46:19.878507 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:19.878478    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:20.287672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.287641    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:46:20.306225 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.306197    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6517163-656a-4391-a298-4c187010beae-kube-api-access-ss2lt\" (UniqueName: \"kubernetes.io/projected/e6517163-656a-4391-a298-4c187010beae-kube-api-access-ss2lt\") pod \"e6517163-656a-4391-a298-4c187010beae\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") "
Oct 31 13:46:20.306423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.306262    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-credential-socket\") pod \"e6517163-656a-4391-a298-4c187010beae\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") "
Oct 31 13:46:20.306423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.306347    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6517163-656a-4391-a298-4c187010beae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6517163-656a-4391-a298-4c187010beae-istio-podinfo\") pod \"e6517163-656a-4391-a298-4c187010beae\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") "
Oct 31 13:46:20.306423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.306398    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-istio-envoy\") pod \"e6517163-656a-4391-a298-4c187010beae\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") "
Oct 31 13:46:20.306607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.306434    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-workload-certs\") pod \"e6517163-656a-4391-a298-4c187010beae\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") "
Oct 31 13:46:20.306607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.306484    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-workload-socket\") pod \"e6517163-656a-4391-a298-4c187010beae\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") "
Oct 31 13:46:20.306607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.306510    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-istio-data\") pod \"e6517163-656a-4391-a298-4c187010beae\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") "
Oct 31 13:46:20.306607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.306539    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6517163-656a-4391-a298-4c187010beae-istio-token\" (UniqueName: \"kubernetes.io/projected/e6517163-656a-4391-a298-4c187010beae-istio-token\") pod \"e6517163-656a-4391-a298-4c187010beae\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") "
Oct 31 13:46:20.306607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.306584    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-credential-socket" pod "e6517163-656a-4391-a298-4c187010beae" (UID: "e6517163-656a-4391-a298-4c187010beae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:20.306936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.306651    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6517163-656a-4391-a298-4c187010beae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6517163-656a-4391-a298-4c187010beae-istiod-ca-cert\") pod \"e6517163-656a-4391-a298-4c187010beae\" (UID: \"e6517163-656a-4391-a298-4c187010beae\") "
Oct 31 13:46:20.306936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.306782    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-workload-certs" pod "e6517163-656a-4391-a298-4c187010beae" (UID: "e6517163-656a-4391-a298-4c187010beae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:20.306936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.306856    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-workload-socket" pod "e6517163-656a-4391-a298-4c187010beae" (UID: "e6517163-656a-4391-a298-4c187010beae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:20.306936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.306900    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-istio-data" pod "e6517163-656a-4391-a298-4c187010beae" (UID: "e6517163-656a-4391-a298-4c187010beae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:20.307296 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.307255    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6517163-656a-4391-a298-4c187010beae-istiod-ca-cert" pod "e6517163-656a-4391-a298-4c187010beae" (UID: "e6517163-656a-4391-a298-4c187010beae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:20.307713 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.307686    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:20.307780 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.307718    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:20.307780 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.307738    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:20.307780 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.307754    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6517163-656a-4391-a298-4c187010beae-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:20.307780 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.307769    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:20.308918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.308896    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-istio-envoy" pod "e6517163-656a-4391-a298-4c187010beae" (UID: "e6517163-656a-4391-a298-4c187010beae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:20.309072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.309047    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6517163-656a-4391-a298-4c187010beae-istio-token" pod "e6517163-656a-4391-a298-4c187010beae" (UID: "e6517163-656a-4391-a298-4c187010beae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:20.309328 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.309305    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6517163-656a-4391-a298-4c187010beae-istio-podinfo" pod "e6517163-656a-4391-a298-4c187010beae" (UID: "e6517163-656a-4391-a298-4c187010beae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:46:20.309328 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.309317    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6517163-656a-4391-a298-4c187010beae-kube-api-access-ss2lt" pod "e6517163-656a-4391-a298-4c187010beae" (UID: "e6517163-656a-4391-a298-4c187010beae"). InnerVolumeSpecName "kube-api-access-ss2lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:20.410320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.410279    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss2lt\" (UniqueName: \"kubernetes.io/projected/e6517163-656a-4391-a298-4c187010beae-kube-api-access-ss2lt\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:20.410320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.410319    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6517163-656a-4391-a298-4c187010beae-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:20.410618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.410334    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6517163-656a-4391-a298-4c187010beae-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:20.410618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.410347    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6517163-656a-4391-a298-4c187010beae-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:20.482498 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.482111    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:20.606913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.606809    4428 generic.go:356] "Generic (PLEG): container finished" podID="e6517163-656a-4391-a298-4c187010beae" containerID="ec70f8a05fea67eac9a8733e166c147282ea548d10bffd295ea779a20c5bd8a8" exitCode=0
Oct 31 13:46:20.606913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.606884    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"
Oct 31 13:46:20.607112 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.606907    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" event={"ID":"e6517163-656a-4391-a298-4c187010beae","Type":"ContainerDied","Data":"ec70f8a05fea67eac9a8733e166c147282ea548d10bffd295ea779a20c5bd8a8"}
Oct 31 13:46:20.607112 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.606965    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj" event={"ID":"e6517163-656a-4391-a298-4c187010beae","Type":"ContainerDied","Data":"99678ff9c2a27787ad66b21b3a5b10ab7acd68290abd52aaffbf7903900b0216"}
Oct 31 13:46:20.607112 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.606988    4428 scope.go:122] "RemoveContainer" containerID="ec70f8a05fea67eac9a8733e166c147282ea548d10bffd295ea779a20c5bd8a8"
Oct 31 13:46:20.607484 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.607453    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:20.608696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.608675    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:46:20.619887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.619852    4428 scope.go:122] "RemoveContainer" containerID="ad4686d66ca13445f4e4dff2af2ef7904296be88a65e415bef969e3a3d22d10c"
Oct 31 13:46:20.631526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.631504    4428 scope.go:122] "RemoveContainer" containerID="ec70f8a05fea67eac9a8733e166c147282ea548d10bffd295ea779a20c5bd8a8"
Oct 31 13:46:20.631892 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:20.631852    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec70f8a05fea67eac9a8733e166c147282ea548d10bffd295ea779a20c5bd8a8\": container with ID starting with ec70f8a05fea67eac9a8733e166c147282ea548d10bffd295ea779a20c5bd8a8 not found: ID does not exist" containerID="ec70f8a05fea67eac9a8733e166c147282ea548d10bffd295ea779a20c5bd8a8"
Oct 31 13:46:20.631955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.631904    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec70f8a05fea67eac9a8733e166c147282ea548d10bffd295ea779a20c5bd8a8"} err="failed to get container status \"ec70f8a05fea67eac9a8733e166c147282ea548d10bffd295ea779a20c5bd8a8\": rpc error: code = NotFound desc = could not find container \"ec70f8a05fea67eac9a8733e166c147282ea548d10bffd295ea779a20c5bd8a8\": container with ID starting with ec70f8a05fea67eac9a8733e166c147282ea548d10bffd295ea779a20c5bd8a8 not found: ID does not exist"
Oct 31 13:46:20.631955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.631931    4428 scope.go:122] "RemoveContainer" containerID="ad4686d66ca13445f4e4dff2af2ef7904296be88a65e415bef969e3a3d22d10c"
Oct 31 13:46:20.632208 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:20.632187    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4686d66ca13445f4e4dff2af2ef7904296be88a65e415bef969e3a3d22d10c\": container with ID starting with ad4686d66ca13445f4e4dff2af2ef7904296be88a65e415bef969e3a3d22d10c not found: ID does not exist" containerID="ad4686d66ca13445f4e4dff2af2ef7904296be88a65e415bef969e3a3d22d10c"
Oct 31 13:46:20.632313 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.632214    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4686d66ca13445f4e4dff2af2ef7904296be88a65e415bef969e3a3d22d10c"} err="failed to get container status \"ad4686d66ca13445f4e4dff2af2ef7904296be88a65e415bef969e3a3d22d10c\": rpc error: code = NotFound desc = could not find container \"ad4686d66ca13445f4e4dff2af2ef7904296be88a65e415bef969e3a3d22d10c\": container with ID starting with ad4686d66ca13445f4e4dff2af2ef7904296be88a65e415bef969e3a3d22d10c not found: ID does not exist"
Oct 31 13:46:20.664805 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.664758    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"]
Oct 31 13:46:20.671073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.671041    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-62778a-openshift-default-6c966fd5b9-s9gxj"]
Oct 31 13:46:20.769985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.769953    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6517163-656a-4391-a298-4c187010beae" path="/var/lib/kubelet/pods/e6517163-656a-4391-a298-4c187010beae/volumes"
Oct 31 13:46:20.794545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:20.794503    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7" podUID="8c73f7d7-3bac-40e8-b386-3742d5a73225" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:46:21.039806 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:21.039774    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:21.228744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:21.228704    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:21.534891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:21.534612    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:21.597584 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:21.597546    4428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680bd885_b3c7_42f5_a3a8_4f2950d0b462.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice/crio-f4ff7173cf463fa77294cb5afd3e25bd0fbc5e211a501b20b208a215eb151f31\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485106e_58f9_4b42_ae34_0136f6345af2.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:46:21.814925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:21.814815    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:22.094763 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.093435    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:22.311892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.306011    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj" podUID="1fb736b6-9425-445b-85bb-be3f5a9598d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:46:22.340890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.340353    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:22.500946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.497459    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"]
Oct 31 13:46:22.500946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.498714    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy"
Oct 31 13:46:22.500946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.498735    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy"
Oct 31 13:46:22.500946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.498757    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy"
Oct 31 13:46:22.500946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.498764    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy"
Oct 31 13:46:22.500946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.498976    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy"
Oct 31 13:46:22.500946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.498991    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6517163-656a-4391-a298-4c187010beae" containerName="istio-proxy"
Oct 31 13:46:22.517953 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.516063    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"
Oct 31 13:46:22.543564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.543530    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"]
Oct 31 13:46:22.726946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.726909    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-612b46-dockercfg-5r8jf\""
Oct 31 13:46:22.727175 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.727011    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-612b46\""
Oct 31 13:46:22.745928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.742964    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"]
Oct 31 13:46:22.877601 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:22.871789    4428 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/ccdd54c42109778317fa4df2b24f94e32ca860293b203e96e4d8470155fef8c5/diff" to get inode usage: stat /var/lib/containers/storage/overlay/ccdd54c42109778317fa4df2b24f94e32ca860293b203e96e4d8470155fef8c5/diff: no such file or directory, extraDiskErr: <nil>
Oct 31 13:46:22.943109 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.943066    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:22.973916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.972132    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtcqf\" (UniqueName: \"kubernetes.io/projected/a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f-kube-api-access-qtcqf\") pod \"payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr\" (UID: \"a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"
Oct 31 13:46:22.973916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.972217    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr\" (UID: \"a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"
Oct 31 13:46:22.974223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.974037    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"
Oct 31 13:46:22.986566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:22.986533    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"]
Oct 31 13:46:23.089986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.082821    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vcwv\" (UniqueName: \"kubernetes.io/projected/ee72f133-ecfd-4910-8026-e0da592ead60-kube-api-access-8vcwv\") pod \"payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996\" (UID: \"ee72f133-ecfd-4910-8026-e0da592ead60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"
Oct 31 13:46:23.089986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.082915    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtcqf\" (UniqueName: \"kubernetes.io/projected/a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f-kube-api-access-qtcqf\") pod \"payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr\" (UID: \"a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"
Oct 31 13:46:23.089986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.082996    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr\" (UID: \"a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"
Oct 31 13:46:23.089986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.083029    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee72f133-ecfd-4910-8026-e0da592ead60-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996\" (UID: \"ee72f133-ecfd-4910-8026-e0da592ead60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"
Oct 31 13:46:23.089986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.086939    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-612b46\""
Oct 31 13:46:23.096843 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.096804    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr\" (UID: \"a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"
Oct 31 13:46:23.099758 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.099726    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtcqf\" (UniqueName: \"kubernetes.io/projected/a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f-kube-api-access-qtcqf\") pod \"payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr\" (UID: \"a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"
Oct 31 13:46:23.109001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.108969    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:23.142252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.142166    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"
Oct 31 13:46:23.184108 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.184071    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vcwv\" (UniqueName: \"kubernetes.io/projected/ee72f133-ecfd-4910-8026-e0da592ead60-kube-api-access-8vcwv\") pod \"payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996\" (UID: \"ee72f133-ecfd-4910-8026-e0da592ead60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"
Oct 31 13:46:23.184463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.184164    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee72f133-ecfd-4910-8026-e0da592ead60-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996\" (UID: \"ee72f133-ecfd-4910-8026-e0da592ead60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"
Oct 31 13:46:23.185109 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.185078    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee72f133-ecfd-4910-8026-e0da592ead60-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996\" (UID: \"ee72f133-ecfd-4910-8026-e0da592ead60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"
Oct 31 13:46:23.205580 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.205547    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vcwv\" (UniqueName: \"kubernetes.io/projected/ee72f133-ecfd-4910-8026-e0da592ead60-kube-api-access-8vcwv\") pod \"payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996\" (UID: \"ee72f133-ecfd-4910-8026-e0da592ead60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"
Oct 31 13:46:23.299277 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.299016    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"
Oct 31 13:46:23.457634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.457437    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"]
Oct 31 13:46:23.462725 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:46:23.462679    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22cfa98_6a24_4aea_89d1_9c3ef6a26f9f.slice/crio-83f3d8a8bd8b84ac6e4cfe74a613caee2cbfcd0baf08f14fea037ab6f09193e7 WatchSource:0}: Error finding container 83f3d8a8bd8b84ac6e4cfe74a613caee2cbfcd0baf08f14fea037ab6f09193e7: Status 404 returned error can't find the container with id 83f3d8a8bd8b84ac6e4cfe74a613caee2cbfcd0baf08f14fea037ab6f09193e7
Oct 31 13:46:23.607401 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:46:23.607369    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee72f133_ecfd_4910_8026_e0da592ead60.slice/crio-83b96b68bedffb70b2ed1cdad743f53369891d1e2115cc75d673a6fce194f6ef WatchSource:0}: Error finding container 83b96b68bedffb70b2ed1cdad743f53369891d1e2115cc75d673a6fce194f6ef: Status 404 returned error can't find the container with id 83b96b68bedffb70b2ed1cdad743f53369891d1e2115cc75d673a6fce194f6ef
Oct 31 13:46:23.607540 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.607418    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"]
Oct 31 13:46:23.910331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.910288    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"]
Oct 31 13:46:23.917249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.916847    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:23.927606 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.927565    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-15dc2f\"/\"kube-root-ca.crt\""
Oct 31 13:46:23.927830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.927620    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-15dc2f\"/\"openshift-service-ca.crt\""
Oct 31 13:46:23.927830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.927723    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-15dc2f\"/\"auth-test-model-15dc2f-kserve-self-signed-certs\""
Oct 31 13:46:23.930243 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.930218    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-15dc2f\"/\"default-dockercfg-6lkgc\""
Oct 31 13:46:23.934536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.934490    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"]
Oct 31 13:46:23.986556 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.986519    4428 generic.go:356] "Generic (PLEG): container finished" podID="1fb736b6-9425-445b-85bb-be3f5a9598d1" containerID="b4e65dadf04e7eb0b0103a88896d02ebdd5495eaec350a62865542013c8f9fa2" exitCode=0
Oct 31 13:46:23.986740 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.986691    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj" event={"ID":"1fb736b6-9425-445b-85bb-be3f5a9598d1","Type":"ContainerDied","Data":"b4e65dadf04e7eb0b0103a88896d02ebdd5495eaec350a62865542013c8f9fa2"}
Oct 31 13:46:23.989491 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.989452    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr" event={"ID":"a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f","Type":"ContainerStarted","Data":"9d360ce6fc453349b02be282d92ec3febea3b2c78a263cf224f3205a2d47b455"}
Oct 31 13:46:23.989646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.989514    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr" event={"ID":"a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f","Type":"ContainerStarted","Data":"83f3d8a8bd8b84ac6e4cfe74a613caee2cbfcd0baf08f14fea037ab6f09193e7"}
Oct 31 13:46:23.989927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.989800    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"
Oct 31 13:46:23.990163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.990129    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"
Oct 31 13:46:23.991612 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.991587    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996" event={"ID":"ee72f133-ecfd-4910-8026-e0da592ead60","Type":"ContainerStarted","Data":"ca30dec0fb80e96c9a4e758e406b69526a568c2e7cf94479b69f1566b60e81c6"}
Oct 31 13:46:23.991718 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.991632    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996" event={"ID":"ee72f133-ecfd-4910-8026-e0da592ead60","Type":"ContainerStarted","Data":"83b96b68bedffb70b2ed1cdad743f53369891d1e2115cc75d673a6fce194f6ef"}
Oct 31 13:46:23.991796 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.991715    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"
Oct 31 13:46:23.999280 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.999250    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-model-cache\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:23.999390 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.999372    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vmrf\" (UniqueName: \"kubernetes.io/projected/2ee33e40-dba2-418c-97f0-f120d64a81f9-kube-api-access-8vmrf\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:23.999708 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.999681    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-dshm\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:23.999859 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.999721    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ee33e40-dba2-418c-97f0-f120d64a81f9-tls-certs\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:23.999998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:23.999978    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-home\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:24.000076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.000025    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-tmp-dir\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:24.023166 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.023064    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr" podStartSLOduration=2.023048879 podStartE2EDuration="2.023048879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:46:24.021489209 +0000 UTC m=+2581.419837844" watchObservedRunningTime="2026-10-31 13:46:24.023048879 +0000 UTC m=+2581.421397508"
Oct 31 13:46:24.037662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.037630    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:24.050454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.050408    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996" podStartSLOduration=2.050395178 podStartE2EDuration="2.050395178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:46:24.04981131 +0000 UTC m=+2581.448159933" watchObservedRunningTime="2026-10-31 13:46:24.050395178 +0000 UTC m=+2581.448743806"
Oct 31 13:46:24.101430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.101399    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb736b6-9425-445b-85bb-be3f5a9598d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fb736b6-9425-445b-85bb-be3f5a9598d1-plugins-config-volume\") pod \"1fb736b6-9425-445b-85bb-be3f5a9598d1\" (UID: \"1fb736b6-9425-445b-85bb-be3f5a9598d1\") "
Oct 31 13:46:24.102015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.101452    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb736b6-9425-445b-85bb-be3f5a9598d1-kube-api-access-w4gzn\" (UniqueName: \"kubernetes.io/projected/1fb736b6-9425-445b-85bb-be3f5a9598d1-kube-api-access-w4gzn\") pod \"1fb736b6-9425-445b-85bb-be3f5a9598d1\" (UID: \"1fb736b6-9425-445b-85bb-be3f5a9598d1\") "
Oct 31 13:46:24.102015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.101930    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-home\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:24.102015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.101931    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb736b6-9425-445b-85bb-be3f5a9598d1-plugins-config-volume" pod "1fb736b6-9425-445b-85bb-be3f5a9598d1" (UID: "1fb736b6-9425-445b-85bb-be3f5a9598d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:24.102208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.102056    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-tmp-dir\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:24.102208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.102199    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-model-cache\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:24.102329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.102282    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vmrf\" (UniqueName: \"kubernetes.io/projected/2ee33e40-dba2-418c-97f0-f120d64a81f9-kube-api-access-8vmrf\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:24.102329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.102293    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-home\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:24.102561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.102538    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-model-cache\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:24.102772 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.102744    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-dshm\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:24.102913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.102789    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ee33e40-dba2-418c-97f0-f120d64a81f9-tls-certs\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:24.102913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.102906    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-tmp-dir\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:24.103048 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.103005    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fb736b6-9425-445b-85bb-be3f5a9598d1-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:24.104954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.104929    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb736b6-9425-445b-85bb-be3f5a9598d1-kube-api-access-w4gzn" pod "1fb736b6-9425-445b-85bb-be3f5a9598d1" (UID: "1fb736b6-9425-445b-85bb-be3f5a9598d1"). InnerVolumeSpecName "kube-api-access-w4gzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:24.105758 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.105731    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-dshm\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:24.107227 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.107201    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ee33e40-dba2-418c-97f0-f120d64a81f9-tls-certs\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:24.118100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.118060    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vmrf\" (UniqueName: \"kubernetes.io/projected/2ee33e40-dba2-418c-97f0-f120d64a81f9-kube-api-access-8vmrf\") pod \"auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") " pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:24.208325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.208270    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4gzn\" (UniqueName: \"kubernetes.io/projected/1fb736b6-9425-445b-85bb-be3f5a9598d1-kube-api-access-w4gzn\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:24.257516 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.255784    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:24.257856 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.257400    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:24.662833 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.662802    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"]
Oct 31 13:46:24.662990 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:46:24.662937    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ee33e40_dba2_418c_97f0_f120d64a81f9.slice/crio-c164131d519c9c53e4c836f56d8d17ca3ca5c4c7e2c166aa74d091f73b7de5f4 WatchSource:0}: Error finding container c164131d519c9c53e4c836f56d8d17ca3ca5c4c7e2c166aa74d091f73b7de5f4: Status 404 returned error can't find the container with id c164131d519c9c53e4c836f56d8d17ca3ca5c4c7e2c166aa74d091f73b7de5f4
Oct 31 13:46:24.781470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.781441    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"]
Oct 31 13:46:24.783567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.781853    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62" podUID="6738ac01-586f-4313-9ced-ccb4fbc5cd39" containerName="maas-api" containerID="cri-o://18865331237ab4c2257453b479895e3fcfca14299f08a55d8b749c9d12118472" gracePeriod=30
Oct 31 13:46:24.906763 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:24.906654    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:25.281499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.281442    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:25.412464 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.412426    4428 generic.go:356] "Generic (PLEG): container finished" podID="6738ac01-586f-4313-9ced-ccb4fbc5cd39" containerID="18865331237ab4c2257453b479895e3fcfca14299f08a55d8b749c9d12118472" exitCode=0
Oct 31 13:46:25.413097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.413062    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62" event={"ID":"6738ac01-586f-4313-9ced-ccb4fbc5cd39","Type":"ContainerDied","Data":"18865331237ab4c2257453b479895e3fcfca14299f08a55d8b749c9d12118472"}
Oct 31 13:46:25.415116 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.415088    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj" event={"ID":"1fb736b6-9425-445b-85bb-be3f5a9598d1","Type":"ContainerDied","Data":"f7444b98e51648e278f0f629ccaab8b2d91b2a179bddb8a283232c27225ba144"}
Oct 31 13:46:25.415301 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.415288    4428 scope.go:122] "RemoveContainer" containerID="b4e65dadf04e7eb0b0103a88896d02ebdd5495eaec350a62865542013c8f9fa2"
Oct 31 13:46:25.415529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.415107    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"
Oct 31 13:46:25.421419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.421387    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q" event={"ID":"2ee33e40-dba2-418c-97f0-f120d64a81f9","Type":"ContainerStarted","Data":"0345a1a15b79c25cc4f30462ccf6cb6c9cc577cf362327dfc387a392c0a38f6d"}
Oct 31 13:46:25.421521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.421449    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q" event={"ID":"2ee33e40-dba2-418c-97f0-f120d64a81f9","Type":"ContainerStarted","Data":"c164131d519c9c53e4c836f56d8d17ca3ca5c4c7e2c166aa74d091f73b7de5f4"}
Oct 31 13:46:25.429018 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.428995    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:46:25.474633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.474566    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q" podStartSLOduration=2.474547444 podStartE2EDuration="2.474547444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:46:25.471004031 +0000 UTC m=+2582.869352655" watchObservedRunningTime="2026-10-31 13:46:25.474547444 +0000 UTC m=+2582.872896068"
Oct 31 13:46:25.502378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.502347    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:25.508059 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.508029    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"]
Oct 31 13:46:25.512536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.512512    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4232c-575c7fd47f-sblsj"]
Oct 31 13:46:25.565647 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.565550    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6738ac01-586f-4313-9ced-ccb4fbc5cd39-kube-api-access-cxr5f\" (UniqueName: \"kubernetes.io/projected/6738ac01-586f-4313-9ced-ccb4fbc5cd39-kube-api-access-cxr5f\") pod \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\" (UID: \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\") "
Oct 31 13:46:25.565647 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.565637    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6738ac01-586f-4313-9ced-ccb4fbc5cd39-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6738ac01-586f-4313-9ced-ccb4fbc5cd39-openshift-service-ca\") pod \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\" (UID: \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\") "
Oct 31 13:46:25.565860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.565694    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6738ac01-586f-4313-9ced-ccb4fbc5cd39-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6738ac01-586f-4313-9ced-ccb4fbc5cd39-metrics-certs\") pod \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\" (UID: \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\") "
Oct 31 13:46:25.566072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.566048    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6738ac01-586f-4313-9ced-ccb4fbc5cd39-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6738ac01-586f-4313-9ced-ccb4fbc5cd39-maas-api-tls\") pod \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\" (UID: \"6738ac01-586f-4313-9ced-ccb4fbc5cd39\") "
Oct 31 13:46:25.566551 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.566521    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6738ac01-586f-4313-9ced-ccb4fbc5cd39-openshift-service-ca" pod "6738ac01-586f-4313-9ced-ccb4fbc5cd39" (UID: "6738ac01-586f-4313-9ced-ccb4fbc5cd39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:25.567244 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.567219    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6738ac01-586f-4313-9ced-ccb4fbc5cd39-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:25.570799 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.570775    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6738ac01-586f-4313-9ced-ccb4fbc5cd39-kube-api-access-cxr5f" pod "6738ac01-586f-4313-9ced-ccb4fbc5cd39" (UID: "6738ac01-586f-4313-9ced-ccb4fbc5cd39"). InnerVolumeSpecName "kube-api-access-cxr5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:25.570986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.570967    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6738ac01-586f-4313-9ced-ccb4fbc5cd39-metrics-certs" pod "6738ac01-586f-4313-9ced-ccb4fbc5cd39" (UID: "6738ac01-586f-4313-9ced-ccb4fbc5cd39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:46:25.572429 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.572400    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6738ac01-586f-4313-9ced-ccb4fbc5cd39-maas-api-tls" pod "6738ac01-586f-4313-9ced-ccb4fbc5cd39" (UID: "6738ac01-586f-4313-9ced-ccb4fbc5cd39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:46:25.709418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.709236    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxr5f\" (UniqueName: \"kubernetes.io/projected/6738ac01-586f-4313-9ced-ccb4fbc5cd39-kube-api-access-cxr5f\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:25.709418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.709289    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6738ac01-586f-4313-9ced-ccb4fbc5cd39-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:25.709418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.709307    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6738ac01-586f-4313-9ced-ccb4fbc5cd39-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:25.710674 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.710632    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:25.905666 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:25.905578    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:26.082895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:26.075971    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:26.261894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:26.260976    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:26.285895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:26.276582    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:26.429625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:26.429591    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"
Oct 31 13:46:26.429625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:26.429606    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62" event={"ID":"6738ac01-586f-4313-9ced-ccb4fbc5cd39","Type":"ContainerDied","Data":"1266e6500e2033531249abd4eb7262c52d1817681dda7db4ec4cf7fa50b23e39"}
Oct 31 13:46:26.429959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:26.429671    4428 scope.go:122] "RemoveContainer" containerID="18865331237ab4c2257453b479895e3fcfca14299f08a55d8b749c9d12118472"
Oct 31 13:46:26.486444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:26.486411    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"]
Oct 31 13:46:26.496382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:26.496350    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4232c-fdff49dfd-9gb62"]
Oct 31 13:46:26.750126 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:26.750092    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:26.759429 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:26.759395    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb736b6-9425-445b-85bb-be3f5a9598d1" path="/var/lib/kubelet/pods/1fb736b6-9425-445b-85bb-be3f5a9598d1/volumes"
Oct 31 13:46:26.760012 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:26.759990    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6738ac01-586f-4313-9ced-ccb4fbc5cd39" path="/var/lib/kubelet/pods/6738ac01-586f-4313-9ced-ccb4fbc5cd39/volumes"
Oct 31 13:46:26.977461 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:26.977408    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:27.260896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.260661    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:27.410604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.410577    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"
Oct 31 13:46:27.506534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.504204    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c73f7d7-3bac-40e8-b386-3742d5a73225-kube-api-access-2fl62\" (UniqueName: \"kubernetes.io/projected/8c73f7d7-3bac-40e8-b386-3742d5a73225-kube-api-access-2fl62\") pod \"8c73f7d7-3bac-40e8-b386-3742d5a73225\" (UID: \"8c73f7d7-3bac-40e8-b386-3742d5a73225\") "
Oct 31 13:46:27.506534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.505768    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c73f7d7-3bac-40e8-b386-3742d5a73225-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c73f7d7-3bac-40e8-b386-3742d5a73225-plugins-config-volume\") pod \"8c73f7d7-3bac-40e8-b386-3742d5a73225\" (UID: \"8c73f7d7-3bac-40e8-b386-3742d5a73225\") "
Oct 31 13:46:27.508691 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.507010    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c73f7d7-3bac-40e8-b386-3742d5a73225-plugins-config-volume" pod "8c73f7d7-3bac-40e8-b386-3742d5a73225" (UID: "8c73f7d7-3bac-40e8-b386-3742d5a73225"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:27.515928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.510917    4428 generic.go:356] "Generic (PLEG): container finished" podID="8c73f7d7-3bac-40e8-b386-3742d5a73225" containerID="d70b97a5f61954d3edfbd4a0196f8032ee138f06a4290d8b8793fcc42836bddb" exitCode=0
Oct 31 13:46:27.515928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.513984    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"
Oct 31 13:46:27.515928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.514568    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7" event={"ID":"8c73f7d7-3bac-40e8-b386-3742d5a73225","Type":"ContainerDied","Data":"d70b97a5f61954d3edfbd4a0196f8032ee138f06a4290d8b8793fcc42836bddb"}
Oct 31 13:46:27.515928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.514673    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7" event={"ID":"8c73f7d7-3bac-40e8-b386-3742d5a73225","Type":"ContainerDied","Data":"c80578ea917c04c320dda3388e6555327339cf6f79a59f44dbc80e3d8b3bc1c7"}
Oct 31 13:46:27.515928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.514724    4428 scope.go:122] "RemoveContainer" containerID="d70b97a5f61954d3edfbd4a0196f8032ee138f06a4290d8b8793fcc42836bddb"
Oct 31 13:46:27.520664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.520027    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c73f7d7-3bac-40e8-b386-3742d5a73225-kube-api-access-2fl62" pod "8c73f7d7-3bac-40e8-b386-3742d5a73225" (UID: "8c73f7d7-3bac-40e8-b386-3742d5a73225"). InnerVolumeSpecName "kube-api-access-2fl62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:27.536520 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.536486    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:27.536607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.536546    4428 scope.go:122] "RemoveContainer" containerID="d70b97a5f61954d3edfbd4a0196f8032ee138f06a4290d8b8793fcc42836bddb"
Oct 31 13:46:27.537056 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:27.537036    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70b97a5f61954d3edfbd4a0196f8032ee138f06a4290d8b8793fcc42836bddb\": container with ID starting with d70b97a5f61954d3edfbd4a0196f8032ee138f06a4290d8b8793fcc42836bddb not found: ID does not exist" containerID="d70b97a5f61954d3edfbd4a0196f8032ee138f06a4290d8b8793fcc42836bddb"
Oct 31 13:46:27.537150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.537068    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70b97a5f61954d3edfbd4a0196f8032ee138f06a4290d8b8793fcc42836bddb"} err="failed to get container status \"d70b97a5f61954d3edfbd4a0196f8032ee138f06a4290d8b8793fcc42836bddb\": rpc error: code = NotFound desc = could not find container \"d70b97a5f61954d3edfbd4a0196f8032ee138f06a4290d8b8793fcc42836bddb\": container with ID starting with d70b97a5f61954d3edfbd4a0196f8032ee138f06a4290d8b8793fcc42836bddb not found: ID does not exist"
Oct 31 13:46:27.608821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.608786    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c73f7d7-3bac-40e8-b386-3742d5a73225-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:27.608821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.608821    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fl62\" (UniqueName: \"kubernetes.io/projected/8c73f7d7-3bac-40e8-b386-3742d5a73225-kube-api-access-2fl62\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:27.847399 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.847370    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"]
Oct 31 13:46:27.853020 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.852990    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62778a-6cd8c4dfc9-xfrs7"]
Oct 31 13:46:27.955744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:27.955713    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:28.443777 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:28.443743    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:28.692309 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:28.692275    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:28.757747 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:28.757716    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c73f7d7-3bac-40e8-b386-3742d5a73225" path="/var/lib/kubelet/pods/8c73f7d7-3bac-40e8-b386-3742d5a73225/volumes"
Oct 31 13:46:29.419351 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:29.419306    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl" podUID="5d8cf8de-bfa2-4e2d-8069-e1b1132fe426" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:46:29.419351 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:29.419339    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t" podUID="530cfbed-6986-4f00-9eb5-bb0d7985827e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:46:29.638024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:29.635440    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:30.283784 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.283742    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"
Oct 31 13:46:30.319655 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.319620    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"
Oct 31 13:46:30.473248 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.473216    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d8cf8de-bfa2-4e2d-8069-e1b1132fe426-kube-api-access-9b4md\" (UniqueName: \"kubernetes.io/projected/5d8cf8de-bfa2-4e2d-8069-e1b1132fe426-kube-api-access-9b4md\") pod \"5d8cf8de-bfa2-4e2d-8069-e1b1132fe426\" (UID: \"5d8cf8de-bfa2-4e2d-8069-e1b1132fe426\") "
Oct 31 13:46:30.473669 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.473647    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530cfbed-6986-4f00-9eb5-bb0d7985827e-kube-api-access-bks5c\" (UniqueName: \"kubernetes.io/projected/530cfbed-6986-4f00-9eb5-bb0d7985827e-kube-api-access-bks5c\") pod \"530cfbed-6986-4f00-9eb5-bb0d7985827e\" (UID: \"530cfbed-6986-4f00-9eb5-bb0d7985827e\") "
Oct 31 13:46:30.473817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.473803    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/530cfbed-6986-4f00-9eb5-bb0d7985827e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530cfbed-6986-4f00-9eb5-bb0d7985827e-plugins-config-volume\") pod \"530cfbed-6986-4f00-9eb5-bb0d7985827e\" (UID: \"530cfbed-6986-4f00-9eb5-bb0d7985827e\") "
Oct 31 13:46:30.474170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.474149    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d8cf8de-bfa2-4e2d-8069-e1b1132fe426-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8cf8de-bfa2-4e2d-8069-e1b1132fe426-plugins-config-volume\") pod \"5d8cf8de-bfa2-4e2d-8069-e1b1132fe426\" (UID: \"5d8cf8de-bfa2-4e2d-8069-e1b1132fe426\") "
Oct 31 13:46:30.477260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.477192    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d8cf8de-bfa2-4e2d-8069-e1b1132fe426-plugins-config-volume" pod "5d8cf8de-bfa2-4e2d-8069-e1b1132fe426" (UID: "5d8cf8de-bfa2-4e2d-8069-e1b1132fe426"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:30.478021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.477984    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:30.478508 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.478365    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530cfbed-6986-4f00-9eb5-bb0d7985827e-plugins-config-volume" pod "530cfbed-6986-4f00-9eb5-bb0d7985827e" (UID: "530cfbed-6986-4f00-9eb5-bb0d7985827e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:30.482640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.482546    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8cf8de-bfa2-4e2d-8069-e1b1132fe426-kube-api-access-9b4md" pod "5d8cf8de-bfa2-4e2d-8069-e1b1132fe426" (UID: "5d8cf8de-bfa2-4e2d-8069-e1b1132fe426"). InnerVolumeSpecName "kube-api-access-9b4md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:30.487533 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.487456    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530cfbed-6986-4f00-9eb5-bb0d7985827e-kube-api-access-bks5c" pod "530cfbed-6986-4f00-9eb5-bb0d7985827e" (UID: "530cfbed-6986-4f00-9eb5-bb0d7985827e"). InnerVolumeSpecName "kube-api-access-bks5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:30.539230 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.539193    4428 generic.go:356] "Generic (PLEG): container finished" podID="5d8cf8de-bfa2-4e2d-8069-e1b1132fe426" containerID="612cee8557ca690e42db9e275a8b012bea92750513f5fd2ab4bb6675d53403b5" exitCode=0
Oct 31 13:46:30.539458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.539272    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"
Oct 31 13:46:30.539458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.539272    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl" event={"ID":"5d8cf8de-bfa2-4e2d-8069-e1b1132fe426","Type":"ContainerDied","Data":"612cee8557ca690e42db9e275a8b012bea92750513f5fd2ab4bb6675d53403b5"}
Oct 31 13:46:30.539458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.539366    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl" event={"ID":"5d8cf8de-bfa2-4e2d-8069-e1b1132fe426","Type":"ContainerDied","Data":"ca14b25de3f8034fe7df2ea8f379609d382e6f05d99650a9b9a10af055f0bd56"}
Oct 31 13:46:30.539458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.539397    4428 scope.go:122] "RemoveContainer" containerID="612cee8557ca690e42db9e275a8b012bea92750513f5fd2ab4bb6675d53403b5"
Oct 31 13:46:30.541231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.541209    4428 generic.go:356] "Generic (PLEG): container finished" podID="530cfbed-6986-4f00-9eb5-bb0d7985827e" containerID="86bb0549b47fbe808d32b79a071312394d443405c1fa078cb82c5ade407dfc90" exitCode=0
Oct 31 13:46:30.541377 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.541302    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"
Oct 31 13:46:30.541545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.541526    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t" event={"ID":"530cfbed-6986-4f00-9eb5-bb0d7985827e","Type":"ContainerDied","Data":"86bb0549b47fbe808d32b79a071312394d443405c1fa078cb82c5ade407dfc90"}
Oct 31 13:46:30.541632 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.541576    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t" event={"ID":"530cfbed-6986-4f00-9eb5-bb0d7985827e","Type":"ContainerDied","Data":"a6b6853ca239c47493ca2904bc583258b1e95fcfb28d92af2ac2772037e4fe2c"}
Oct 31 13:46:30.560244 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.560213    4428 scope.go:122] "RemoveContainer" containerID="612cee8557ca690e42db9e275a8b012bea92750513f5fd2ab4bb6675d53403b5"
Oct 31 13:46:30.560630 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:30.560610    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"612cee8557ca690e42db9e275a8b012bea92750513f5fd2ab4bb6675d53403b5\": container with ID starting with 612cee8557ca690e42db9e275a8b012bea92750513f5fd2ab4bb6675d53403b5 not found: ID does not exist" containerID="612cee8557ca690e42db9e275a8b012bea92750513f5fd2ab4bb6675d53403b5"
Oct 31 13:46:30.560706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.560641    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"612cee8557ca690e42db9e275a8b012bea92750513f5fd2ab4bb6675d53403b5"} err="failed to get container status \"612cee8557ca690e42db9e275a8b012bea92750513f5fd2ab4bb6675d53403b5\": rpc error: code = NotFound desc = could not find container \"612cee8557ca690e42db9e275a8b012bea92750513f5fd2ab4bb6675d53403b5\": container with ID starting with 612cee8557ca690e42db9e275a8b012bea92750513f5fd2ab4bb6675d53403b5 not found: ID does not exist"
Oct 31 13:46:30.560706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.560661    4428 scope.go:122] "RemoveContainer" containerID="86bb0549b47fbe808d32b79a071312394d443405c1fa078cb82c5ade407dfc90"
Oct 31 13:46:30.573713 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.573690    4428 scope.go:122] "RemoveContainer" containerID="86bb0549b47fbe808d32b79a071312394d443405c1fa078cb82c5ade407dfc90"
Oct 31 13:46:30.574180 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:30.574148    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86bb0549b47fbe808d32b79a071312394d443405c1fa078cb82c5ade407dfc90\": container with ID starting with 86bb0549b47fbe808d32b79a071312394d443405c1fa078cb82c5ade407dfc90 not found: ID does not exist" containerID="86bb0549b47fbe808d32b79a071312394d443405c1fa078cb82c5ade407dfc90"
Oct 31 13:46:30.574323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.574193    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86bb0549b47fbe808d32b79a071312394d443405c1fa078cb82c5ade407dfc90"} err="failed to get container status \"86bb0549b47fbe808d32b79a071312394d443405c1fa078cb82c5ade407dfc90\": rpc error: code = NotFound desc = could not find container \"86bb0549b47fbe808d32b79a071312394d443405c1fa078cb82c5ade407dfc90\": container with ID starting with 86bb0549b47fbe808d32b79a071312394d443405c1fa078cb82c5ade407dfc90 not found: ID does not exist"
Oct 31 13:46:30.574971 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.574945    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"]
Oct 31 13:46:30.577334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.577304    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b4md\" (UniqueName: \"kubernetes.io/projected/5d8cf8de-bfa2-4e2d-8069-e1b1132fe426-kube-api-access-9b4md\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:30.577444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.577335    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bks5c\" (UniqueName: \"kubernetes.io/projected/530cfbed-6986-4f00-9eb5-bb0d7985827e-kube-api-access-bks5c\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:30.577444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.577347    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530cfbed-6986-4f00-9eb5-bb0d7985827e-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:30.577444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.577356    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d8cf8de-bfa2-4e2d-8069-e1b1132fe426-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:30.581916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.581891    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-61da9d-657cdf778c-cr5gl"]
Oct 31 13:46:30.594779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.594740    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"]
Oct 31 13:46:30.600667 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.600641    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-61da9d-75c8647c9-b7t8t"]
Oct 31 13:46:30.758626 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.758544    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530cfbed-6986-4f00-9eb5-bb0d7985827e" path="/var/lib/kubelet/pods/530cfbed-6986-4f00-9eb5-bb0d7985827e/volumes"
Oct 31 13:46:30.759106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:30.759090    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d8cf8de-bfa2-4e2d-8069-e1b1132fe426" path="/var/lib/kubelet/pods/5d8cf8de-bfa2-4e2d-8069-e1b1132fe426/volumes"
Oct 31 13:46:31.417897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:31.415225    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:32.217900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:32.210848    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:33.088967 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:33.088509    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:33.490662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:33.481644    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:33.660367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:33.660333    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:33.741811 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:33.741706    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"]
Oct 31 13:46:33.742056 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:33.741987    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n" podUID="e0b75287-cf60-49fe-8b06-dec5a12b5cd1" containerName="istio-proxy" containerID="cri-o://d9479088ff99629a17781a69a3faf52cd9210756030aab30452ace9fc85aef59" gracePeriod=30
Oct 31 13:46:33.748849 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:33.748824    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"]
Oct 31 13:46:33.749046 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:33.749025    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy" containerID="cri-o://5772b0da2ef97500dc3b575bed722d9e518fb240c624309f6e14b9c5dbba38c8" gracePeriod=30
Oct 31 13:46:33.853294 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:33.853258    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:34.045190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:34.045099    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:34.257616 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:34.257585    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:34.273885 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:34.273848    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:34.588380 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:34.588279    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:46:35.221895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:35.215705    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:35.425113 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:35.425080    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"
Oct 31 13:46:35.425332 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:35.425175    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"
Oct 31 13:46:35.883423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:35.883388    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:36.050464 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:36.050426    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:36.174081 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:36.173973    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"]
Oct 31 13:46:36.174418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:36.174370    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww" podUID="d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" containerName="istio-proxy" containerID="cri-o://903a102639cbd6fffd6ee7f723cb568f6dc40343b4bc6f4b31fbce3da61165e6" gracePeriod=30
Oct 31 13:46:36.183755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:36.183719    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"]
Oct 31 13:46:36.184090 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:36.184054    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6" podUID="93007950-2d7f-495f-a37d-c8729493b79d" containerName="istio-proxy" containerID="cri-o://1d0fc13c282706c183153e71f2b0ac0c3e7c9183b0455f5df2f40aca13d6f89d" gracePeriod=30
Oct 31 13:46:36.285910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:36.283714    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:37.288027 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:37.284209    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:37.599171 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:37.599094    4428 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:46:37.599171 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:37.599157    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww" podUID="d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:46:37.962611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:37.960785    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"]
Oct 31 13:46:37.962611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:37.961058    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk" podUID="1e0031a9-bab3-4590-8add-4eeafdf4ac52" containerName="maas-api" containerID="cri-o://2db241a630f4956776f1179fb7d539350154dafeb4d37da4a750a2e0ebc78b26" gracePeriod=30
Oct 31 13:46:38.070263 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.070224    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:38.504894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.498256    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:38.520822 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.520785    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:46:38.618759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.618670    4428 generic.go:356] "Generic (PLEG): container finished" podID="1e0031a9-bab3-4590-8add-4eeafdf4ac52" containerID="2db241a630f4956776f1179fb7d539350154dafeb4d37da4a750a2e0ebc78b26" exitCode=0
Oct 31 13:46:38.618759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.618714    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"
Oct 31 13:46:38.619048 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.618751    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk" event={"ID":"1e0031a9-bab3-4590-8add-4eeafdf4ac52","Type":"ContainerDied","Data":"2db241a630f4956776f1179fb7d539350154dafeb4d37da4a750a2e0ebc78b26"}
Oct 31 13:46:38.619048 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.618806    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk" event={"ID":"1e0031a9-bab3-4590-8add-4eeafdf4ac52","Type":"ContainerDied","Data":"3e89206053d6b87efd82097b8dcd2301cc467c5b8e6deeb51e95c2d9f9b70d3b"}
Oct 31 13:46:38.619048 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.618829    4428 scope.go:122] "RemoveContainer" containerID="2db241a630f4956776f1179fb7d539350154dafeb4d37da4a750a2e0ebc78b26"
Oct 31 13:46:38.630139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.630119    4428 scope.go:122] "RemoveContainer" containerID="2db241a630f4956776f1179fb7d539350154dafeb4d37da4a750a2e0ebc78b26"
Oct 31 13:46:38.630470 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:38.630447    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db241a630f4956776f1179fb7d539350154dafeb4d37da4a750a2e0ebc78b26\": container with ID starting with 2db241a630f4956776f1179fb7d539350154dafeb4d37da4a750a2e0ebc78b26 not found: ID does not exist" containerID="2db241a630f4956776f1179fb7d539350154dafeb4d37da4a750a2e0ebc78b26"
Oct 31 13:46:38.630515 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.630480    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db241a630f4956776f1179fb7d539350154dafeb4d37da4a750a2e0ebc78b26"} err="failed to get container status \"2db241a630f4956776f1179fb7d539350154dafeb4d37da4a750a2e0ebc78b26\": rpc error: code = NotFound desc = could not find container \"2db241a630f4956776f1179fb7d539350154dafeb4d37da4a750a2e0ebc78b26\": container with ID starting with 2db241a630f4956776f1179fb7d539350154dafeb4d37da4a750a2e0ebc78b26 not found: ID does not exist"
Oct 31 13:46:38.690907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.689421    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e0031a9-bab3-4590-8add-4eeafdf4ac52-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0031a9-bab3-4590-8add-4eeafdf4ac52-metrics-certs\") pod \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\" (UID: \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\") "
Oct 31 13:46:38.690907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.689552    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e0031a9-bab3-4590-8add-4eeafdf4ac52-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0031a9-bab3-4590-8add-4eeafdf4ac52-openshift-service-ca\") pod \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\" (UID: \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\") "
Oct 31 13:46:38.690907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.689706    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0031a9-bab3-4590-8add-4eeafdf4ac52-kube-api-access-fk2r5\" (UniqueName: \"kubernetes.io/projected/1e0031a9-bab3-4590-8add-4eeafdf4ac52-kube-api-access-fk2r5\") pod \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\" (UID: \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\") "
Oct 31 13:46:38.690907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.689792    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e0031a9-bab3-4590-8add-4eeafdf4ac52-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0031a9-bab3-4590-8add-4eeafdf4ac52-maas-api-tls\") pod \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\" (UID: \"1e0031a9-bab3-4590-8add-4eeafdf4ac52\") "
Oct 31 13:46:38.699915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.692961    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:38.699915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.696739    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0031a9-bab3-4590-8add-4eeafdf4ac52-openshift-service-ca" pod "1e0031a9-bab3-4590-8add-4eeafdf4ac52" (UID: "1e0031a9-bab3-4590-8add-4eeafdf4ac52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:38.717153 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.714781    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e0031a9-bab3-4590-8add-4eeafdf4ac52-metrics-certs" pod "1e0031a9-bab3-4590-8add-4eeafdf4ac52" (UID: "1e0031a9-bab3-4590-8add-4eeafdf4ac52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:46:38.717153 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.715429    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0031a9-bab3-4590-8add-4eeafdf4ac52-kube-api-access-fk2r5" pod "1e0031a9-bab3-4590-8add-4eeafdf4ac52" (UID: "1e0031a9-bab3-4590-8add-4eeafdf4ac52"). InnerVolumeSpecName "kube-api-access-fk2r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:38.718632 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.718579    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e0031a9-bab3-4590-8add-4eeafdf4ac52-maas-api-tls" pod "1e0031a9-bab3-4590-8add-4eeafdf4ac52" (UID: "1e0031a9-bab3-4590-8add-4eeafdf4ac52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:46:38.791975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.791940    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0031a9-bab3-4590-8add-4eeafdf4ac52-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:38.791975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.791971    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk2r5\" (UniqueName: \"kubernetes.io/projected/1e0031a9-bab3-4590-8add-4eeafdf4ac52-kube-api-access-fk2r5\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:38.792211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.791987    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0031a9-bab3-4590-8add-4eeafdf4ac52-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:38.792211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.791999    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0031a9-bab3-4590-8add-4eeafdf4ac52-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:38.860248 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.860218    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:38.964365 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.964329    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"]
Oct 31 13:46:38.971059 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:38.971024    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4fde10-5b6cc8847f-zrmpk"]
Oct 31 13:46:39.064347 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.064319    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:39.077013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.076977    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:46:39.126599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.126523    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:46:39.198411 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.198370    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-data\") pod \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") "
Oct 31 13:46:39.198644 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.198476    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-workload-certs\") pod \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") "
Oct 31 13:46:39.198644 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.198560    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-podinfo\") pod \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") "
Oct 31 13:46:39.198644 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.198586    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-token\" (UniqueName: \"kubernetes.io/projected/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-token\") pod \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") "
Oct 31 13:46:39.199006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.198690    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-envoy\") pod \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") "
Oct 31 13:46:39.199461 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.199430    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-credential-socket\") pod \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") "
Oct 31 13:46:39.199575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.199552    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istiod-ca-cert\") pod \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") "
Oct 31 13:46:39.199639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.199613    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-kube-api-access-bfg8b\" (UniqueName: \"kubernetes.io/projected/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-kube-api-access-bfg8b\") pod \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") "
Oct 31 13:46:39.199717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.199700    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-workload-socket\") pod \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\" (UID: \"e0b75287-cf60-49fe-8b06-dec5a12b5cd1\") "
Oct 31 13:46:39.200235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.200119    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-workload-certs" pod "e0b75287-cf60-49fe-8b06-dec5a12b5cd1" (UID: "e0b75287-cf60-49fe-8b06-dec5a12b5cd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.200472 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.200445    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istiod-ca-cert" pod "e0b75287-cf60-49fe-8b06-dec5a12b5cd1" (UID: "e0b75287-cf60-49fe-8b06-dec5a12b5cd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:39.200558 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.200528    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-credential-socket" pod "e0b75287-cf60-49fe-8b06-dec5a12b5cd1" (UID: "e0b75287-cf60-49fe-8b06-dec5a12b5cd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.200801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.200786    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-workload-socket" pod "e0b75287-cf60-49fe-8b06-dec5a12b5cd1" (UID: "e0b75287-cf60-49fe-8b06-dec5a12b5cd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.200988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.200966    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-data" pod "e0b75287-cf60-49fe-8b06-dec5a12b5cd1" (UID: "e0b75287-cf60-49fe-8b06-dec5a12b5cd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.201371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.201355    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.201425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.201375    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.201425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.201384    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.201425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.201393    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.201425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.201402    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.202857 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.202817    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-kube-api-access-bfg8b" pod "e0b75287-cf60-49fe-8b06-dec5a12b5cd1" (UID: "e0b75287-cf60-49fe-8b06-dec5a12b5cd1"). InnerVolumeSpecName "kube-api-access-bfg8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:39.208301 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.208262    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-envoy" pod "e0b75287-cf60-49fe-8b06-dec5a12b5cd1" (UID: "e0b75287-cf60-49fe-8b06-dec5a12b5cd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.209741 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.209715    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-podinfo" pod "e0b75287-cf60-49fe-8b06-dec5a12b5cd1" (UID: "e0b75287-cf60-49fe-8b06-dec5a12b5cd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:46:39.209930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.209904    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-token" pod "e0b75287-cf60-49fe-8b06-dec5a12b5cd1" (UID: "e0b75287-cf60-49fe-8b06-dec5a12b5cd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:39.302183 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.302149    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a9f039-bd18-4c3a-88a3-973bc6721086-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0a9f039-bd18-4c3a-88a3-973bc6721086-istiod-ca-cert\") pod \"f0a9f039-bd18-4c3a-88a3-973bc6721086\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") "
Oct 31 13:46:39.302405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.302195    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-workload-certs\") pod \"f0a9f039-bd18-4c3a-88a3-973bc6721086\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") "
Oct 31 13:46:39.302405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.302261    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-credential-socket\") pod \"f0a9f039-bd18-4c3a-88a3-973bc6721086\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") "
Oct 31 13:46:39.302405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.302329    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-data\") pod \"f0a9f039-bd18-4c3a-88a3-973bc6721086\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") "
Oct 31 13:46:39.302405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.302375    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-envoy\") pod \"f0a9f039-bd18-4c3a-88a3-973bc6721086\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") "
Oct 31 13:46:39.302654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.302412    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-token\" (UniqueName: \"kubernetes.io/projected/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-token\") pod \"f0a9f039-bd18-4c3a-88a3-973bc6721086\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") "
Oct 31 13:46:39.302654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.302491    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-workload-socket\") pod \"f0a9f039-bd18-4c3a-88a3-973bc6721086\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") "
Oct 31 13:46:39.302654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.302622    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-workload-certs" pod "f0a9f039-bd18-4c3a-88a3-973bc6721086" (UID: "f0a9f039-bd18-4c3a-88a3-973bc6721086"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.302817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.302655    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-credential-socket" pod "f0a9f039-bd18-4c3a-88a3-973bc6721086" (UID: "f0a9f039-bd18-4c3a-88a3-973bc6721086"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.302817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.302662    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-podinfo\") pod \"f0a9f039-bd18-4c3a-88a3-973bc6721086\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") "
Oct 31 13:46:39.302817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.302723    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a9f039-bd18-4c3a-88a3-973bc6721086-istiod-ca-cert" pod "f0a9f039-bd18-4c3a-88a3-973bc6721086" (UID: "f0a9f039-bd18-4c3a-88a3-973bc6721086"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:39.303006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.302821    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a9f039-bd18-4c3a-88a3-973bc6721086-kube-api-access-pgqhk\" (UniqueName: \"kubernetes.io/projected/f0a9f039-bd18-4c3a-88a3-973bc6721086-kube-api-access-pgqhk\") pod \"f0a9f039-bd18-4c3a-88a3-973bc6721086\" (UID: \"f0a9f039-bd18-4c3a-88a3-973bc6721086\") "
Oct 31 13:46:39.303006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.302845    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-data" pod "f0a9f039-bd18-4c3a-88a3-973bc6721086" (UID: "f0a9f039-bd18-4c3a-88a3-973bc6721086"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.303006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.302928    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-workload-socket" pod "f0a9f039-bd18-4c3a-88a3-973bc6721086" (UID: "f0a9f039-bd18-4c3a-88a3-973bc6721086"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.303850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.303819    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.304039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.303854    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.304039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.303890    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.304039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.303907    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfg8b\" (UniqueName: \"kubernetes.io/projected/e0b75287-cf60-49fe-8b06-dec5a12b5cd1-kube-api-access-bfg8b\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.304039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.303922    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0a9f039-bd18-4c3a-88a3-973bc6721086-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.304039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.303937    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.304039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.303953    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.304039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.303968    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.304039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.303982    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.304924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.304899    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-token" pod "f0a9f039-bd18-4c3a-88a3-973bc6721086" (UID: "f0a9f039-bd18-4c3a-88a3-973bc6721086"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:39.305033 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.304935    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-envoy" pod "f0a9f039-bd18-4c3a-88a3-973bc6721086" (UID: "f0a9f039-bd18-4c3a-88a3-973bc6721086"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:39.305033 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.304942    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-podinfo" pod "f0a9f039-bd18-4c3a-88a3-973bc6721086" (UID: "f0a9f039-bd18-4c3a-88a3-973bc6721086"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:46:39.305318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.305301    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a9f039-bd18-4c3a-88a3-973bc6721086-kube-api-access-pgqhk" pod "f0a9f039-bd18-4c3a-88a3-973bc6721086" (UID: "f0a9f039-bd18-4c3a-88a3-973bc6721086"). InnerVolumeSpecName "kube-api-access-pgqhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:39.405171 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.405087    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.405171 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.405117    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgqhk\" (UniqueName: \"kubernetes.io/projected/f0a9f039-bd18-4c3a-88a3-973bc6721086-kube-api-access-pgqhk\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.405171 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.405128    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.405171 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.405137    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0a9f039-bd18-4c3a-88a3-973bc6721086-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:39.628776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.628731    4428 generic.go:356] "Generic (PLEG): container finished" podID="e0b75287-cf60-49fe-8b06-dec5a12b5cd1" containerID="d9479088ff99629a17781a69a3faf52cd9210756030aab30452ace9fc85aef59" exitCode=0
Oct 31 13:46:39.628776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.628778    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"
Oct 31 13:46:39.629364 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.628820    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n" event={"ID":"e0b75287-cf60-49fe-8b06-dec5a12b5cd1","Type":"ContainerDied","Data":"d9479088ff99629a17781a69a3faf52cd9210756030aab30452ace9fc85aef59"}
Oct 31 13:46:39.629364 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.628904    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n" event={"ID":"e0b75287-cf60-49fe-8b06-dec5a12b5cd1","Type":"ContainerDied","Data":"1023d7432271b97af10429eb4313b8bfac0a2fd62994c87450ea0f385fd5881f"}
Oct 31 13:46:39.629364 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.628936    4428 scope.go:122] "RemoveContainer" containerID="d9479088ff99629a17781a69a3faf52cd9210756030aab30452ace9fc85aef59"
Oct 31 13:46:39.633338 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.632413    4428 generic.go:356] "Generic (PLEG): container finished" podID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerID="5772b0da2ef97500dc3b575bed722d9e518fb240c624309f6e14b9c5dbba38c8" exitCode=0
Oct 31 13:46:39.633338 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.632448    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" event={"ID":"f0a9f039-bd18-4c3a-88a3-973bc6721086","Type":"ContainerDied","Data":"5772b0da2ef97500dc3b575bed722d9e518fb240c624309f6e14b9c5dbba38c8"}
Oct 31 13:46:39.633338 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.632517    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k" event={"ID":"f0a9f039-bd18-4c3a-88a3-973bc6721086","Type":"ContainerDied","Data":"06f38a239f9cb3834be8b129d720bd76f6700a35253118ce727370f28774c14e"}
Oct 31 13:46:39.633338 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.632538    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"
Oct 31 13:46:39.660417 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.660383    4428 scope.go:122] "RemoveContainer" containerID="d9479088ff99629a17781a69a3faf52cd9210756030aab30452ace9fc85aef59"
Oct 31 13:46:39.664280 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:39.664240    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9479088ff99629a17781a69a3faf52cd9210756030aab30452ace9fc85aef59\": container with ID starting with d9479088ff99629a17781a69a3faf52cd9210756030aab30452ace9fc85aef59 not found: ID does not exist" containerID="d9479088ff99629a17781a69a3faf52cd9210756030aab30452ace9fc85aef59"
Oct 31 13:46:39.664423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.664293    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9479088ff99629a17781a69a3faf52cd9210756030aab30452ace9fc85aef59"} err="failed to get container status \"d9479088ff99629a17781a69a3faf52cd9210756030aab30452ace9fc85aef59\": rpc error: code = NotFound desc = could not find container \"d9479088ff99629a17781a69a3faf52cd9210756030aab30452ace9fc85aef59\": container with ID starting with d9479088ff99629a17781a69a3faf52cd9210756030aab30452ace9fc85aef59 not found: ID does not exist"
Oct 31 13:46:39.664423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.664322    4428 scope.go:122] "RemoveContainer" containerID="5772b0da2ef97500dc3b575bed722d9e518fb240c624309f6e14b9c5dbba38c8"
Oct 31 13:46:39.684008 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.683979    4428 scope.go:122] "RemoveContainer" containerID="1e26e91b65e5cb9ec7273df8e235af5f5b11d66e27279d7c5205d3e2cf060432"
Oct 31 13:46:39.696569 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.696534    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"]
Oct 31 13:46:39.708494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.708462    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-ggv6k"]
Oct 31 13:46:39.708688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.708660    4428 scope.go:122] "RemoveContainer" containerID="5772b0da2ef97500dc3b575bed722d9e518fb240c624309f6e14b9c5dbba38c8"
Oct 31 13:46:39.710406 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:39.710380    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5772b0da2ef97500dc3b575bed722d9e518fb240c624309f6e14b9c5dbba38c8\": container with ID starting with 5772b0da2ef97500dc3b575bed722d9e518fb240c624309f6e14b9c5dbba38c8 not found: ID does not exist" containerID="5772b0da2ef97500dc3b575bed722d9e518fb240c624309f6e14b9c5dbba38c8"
Oct 31 13:46:39.710534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.710413    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5772b0da2ef97500dc3b575bed722d9e518fb240c624309f6e14b9c5dbba38c8"} err="failed to get container status \"5772b0da2ef97500dc3b575bed722d9e518fb240c624309f6e14b9c5dbba38c8\": rpc error: code = NotFound desc = could not find container \"5772b0da2ef97500dc3b575bed722d9e518fb240c624309f6e14b9c5dbba38c8\": container with ID starting with 5772b0da2ef97500dc3b575bed722d9e518fb240c624309f6e14b9c5dbba38c8 not found: ID does not exist"
Oct 31 13:46:39.710534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.710434    4428 scope.go:122] "RemoveContainer" containerID="1e26e91b65e5cb9ec7273df8e235af5f5b11d66e27279d7c5205d3e2cf060432"
Oct 31 13:46:39.710762 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:39.710742    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e26e91b65e5cb9ec7273df8e235af5f5b11d66e27279d7c5205d3e2cf060432\": container with ID starting with 1e26e91b65e5cb9ec7273df8e235af5f5b11d66e27279d7c5205d3e2cf060432 not found: ID does not exist" containerID="1e26e91b65e5cb9ec7273df8e235af5f5b11d66e27279d7c5205d3e2cf060432"
Oct 31 13:46:39.710823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.710769    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e26e91b65e5cb9ec7273df8e235af5f5b11d66e27279d7c5205d3e2cf060432"} err="failed to get container status \"1e26e91b65e5cb9ec7273df8e235af5f5b11d66e27279d7c5205d3e2cf060432\": rpc error: code = NotFound desc = could not find container \"1e26e91b65e5cb9ec7273df8e235af5f5b11d66e27279d7c5205d3e2cf060432\": container with ID starting with 1e26e91b65e5cb9ec7273df8e235af5f5b11d66e27279d7c5205d3e2cf060432 not found: ID does not exist"
Oct 31 13:46:39.728065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.728036    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"]
Oct 31 13:46:39.739526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.739496    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-hgz4n"]
Oct 31 13:46:39.808888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:39.806672    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:40.803776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:40.803742    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0031a9-bab3-4590-8add-4eeafdf4ac52" path="/var/lib/kubelet/pods/1e0031a9-bab3-4590-8add-4eeafdf4ac52/volumes"
Oct 31 13:46:40.804537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:40.804515    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b75287-cf60-49fe-8b06-dec5a12b5cd1" path="/var/lib/kubelet/pods/e0b75287-cf60-49fe-8b06-dec5a12b5cd1/volumes"
Oct 31 13:46:40.805042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:40.805024    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" path="/var/lib/kubelet/pods/f0a9f039-bd18-4c3a-88a3-973bc6721086/volumes"
Oct 31 13:46:40.819417 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:40.819382    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:41.126335 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.126242    4428 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:46:41.126335 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.126306    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6" podUID="93007950-2d7f-495f-a37d-c8729493b79d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:46:41.510573 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.510532    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:46:41.554071 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.554043    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:46:41.655768 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.655726    4428 generic.go:356] "Generic (PLEG): container finished" podID="93007950-2d7f-495f-a37d-c8729493b79d" containerID="1d0fc13c282706c183153e71f2b0ac0c3e7c9183b0455f5df2f40aca13d6f89d" exitCode=0
Oct 31 13:46:41.656084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.655797    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"
Oct 31 13:46:41.656084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.655979    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6" event={"ID":"93007950-2d7f-495f-a37d-c8729493b79d","Type":"ContainerDied","Data":"1d0fc13c282706c183153e71f2b0ac0c3e7c9183b0455f5df2f40aca13d6f89d"}
Oct 31 13:46:41.656084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.656045    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6" event={"ID":"93007950-2d7f-495f-a37d-c8729493b79d","Type":"ContainerDied","Data":"bcd7791a5aed6ab1b122797c6ae22acf7119344bd57cd0d8fa78eedc77be77ff"}
Oct 31 13:46:41.656271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.656116    4428 scope.go:122] "RemoveContainer" containerID="1d0fc13c282706c183153e71f2b0ac0c3e7c9183b0455f5df2f40aca13d6f89d"
Oct 31 13:46:41.659985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.659956    4428 generic.go:356] "Generic (PLEG): container finished" podID="d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" containerID="903a102639cbd6fffd6ee7f723cb568f6dc40343b4bc6f4b31fbce3da61165e6" exitCode=0
Oct 31 13:46:41.660230 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.660200    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww" event={"ID":"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954","Type":"ContainerDied","Data":"903a102639cbd6fffd6ee7f723cb568f6dc40343b4bc6f4b31fbce3da61165e6"}
Oct 31 13:46:41.662799 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.660284    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww" event={"ID":"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954","Type":"ContainerDied","Data":"f831ff3186b5c94d85d2445d7e17bec7f0785ca74cdeb3f0d17cf27df301a625"}
Oct 31 13:46:41.662799 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.660420    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"
Oct 31 13:46:41.663976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.661940    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-data\") pod \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") "
Oct 31 13:46:41.663976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.663426    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93007950-2d7f-495f-a37d-c8729493b79d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93007950-2d7f-495f-a37d-c8729493b79d-istio-podinfo\") pod \"93007950-2d7f-495f-a37d-c8729493b79d\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") "
Oct 31 13:46:41.663976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.663544    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-workload-certs\") pod \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") "
Oct 31 13:46:41.663976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.663608    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93007950-2d7f-495f-a37d-c8729493b79d-istio-token\" (UniqueName: \"kubernetes.io/projected/93007950-2d7f-495f-a37d-c8729493b79d-istio-token\") pod \"93007950-2d7f-495f-a37d-c8729493b79d\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") "
Oct 31 13:46:41.664646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.663726    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93007950-2d7f-495f-a37d-c8729493b79d-kube-api-access-bq65m\" (UniqueName: \"kubernetes.io/projected/93007950-2d7f-495f-a37d-c8729493b79d-kube-api-access-bq65m\") pod \"93007950-2d7f-495f-a37d-c8729493b79d\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") "
Oct 31 13:46:41.664646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.664412    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93007950-2d7f-495f-a37d-c8729493b79d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93007950-2d7f-495f-a37d-c8729493b79d-istiod-ca-cert\") pod \"93007950-2d7f-495f-a37d-c8729493b79d\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") "
Oct 31 13:46:41.664646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.664494    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-kube-api-access-4lkqp\" (UniqueName: \"kubernetes.io/projected/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-kube-api-access-4lkqp\") pod \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") "
Oct 31 13:46:41.664646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.664549    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-token\" (UniqueName: \"kubernetes.io/projected/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-token\") pod \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") "
Oct 31 13:46:41.664646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.664630    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istiod-ca-cert\") pod \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") "
Oct 31 13:46:41.664951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.664663    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-credential-socket\") pod \"93007950-2d7f-495f-a37d-c8729493b79d\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") "
Oct 31 13:46:41.664951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.664724    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-istio-envoy\") pod \"93007950-2d7f-495f-a37d-c8729493b79d\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") "
Oct 31 13:46:41.664951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.664778    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-workload-socket\") pod \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") "
Oct 31 13:46:41.664951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.664827    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-workload-socket\") pod \"93007950-2d7f-495f-a37d-c8729493b79d\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") "
Oct 31 13:46:41.665184 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.665131    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-credential-socket\") pod \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") "
Oct 31 13:46:41.665325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.665306    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-envoy\") pod \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") "
Oct 31 13:46:41.665397 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.665342    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-istio-data\") pod \"93007950-2d7f-495f-a37d-c8729493b79d\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") "
Oct 31 13:46:41.665397 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.665386    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-workload-certs\") pod \"93007950-2d7f-495f-a37d-c8729493b79d\" (UID: \"93007950-2d7f-495f-a37d-c8729493b79d\") "
Oct 31 13:46:41.666776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.666752    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-podinfo\") pod \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\" (UID: \"d5efe5a3-93a2-4e34-ba6d-1037ac9b0954\") "
Oct 31 13:46:41.668268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.668228    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-workload-socket" pod "93007950-2d7f-495f-a37d-c8729493b79d" (UID: "93007950-2d7f-495f-a37d-c8729493b79d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.668486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.668462    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93007950-2d7f-495f-a37d-c8729493b79d-istiod-ca-cert" pod "93007950-2d7f-495f-a37d-c8729493b79d" (UID: "93007950-2d7f-495f-a37d-c8729493b79d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:41.668575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.668524    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istiod-ca-cert" pod "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" (UID: "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:41.668820 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.668797    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-credential-socket" pod "93007950-2d7f-495f-a37d-c8729493b79d" (UID: "93007950-2d7f-495f-a37d-c8729493b79d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.670327 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.670294    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-token" pod "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" (UID: "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:41.670431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.670378    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-podinfo" pod "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" (UID: "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:46:41.670641 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.670616    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-istio-data" pod "93007950-2d7f-495f-a37d-c8729493b79d" (UID: "93007950-2d7f-495f-a37d-c8729493b79d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.670769 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.670742    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-workload-socket" pod "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" (UID: "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.670996 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.670971    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-workload-certs" pod "93007950-2d7f-495f-a37d-c8729493b79d" (UID: "93007950-2d7f-495f-a37d-c8729493b79d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.671173 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.671154    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-workload-certs" pod "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" (UID: "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.671357 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.671337    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-credential-socket" pod "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" (UID: "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.671546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.671525    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-data" pod "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" (UID: "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.675045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.674630    4428 scope.go:122] "RemoveContainer" containerID="1d0fc13c282706c183153e71f2b0ac0c3e7c9183b0455f5df2f40aca13d6f89d"
Oct 31 13:46:41.675045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.674825    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-envoy" pod "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" (UID: "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.675045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.674954    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93007950-2d7f-495f-a37d-c8729493b79d-kube-api-access-bq65m" pod "93007950-2d7f-495f-a37d-c8729493b79d" (UID: "93007950-2d7f-495f-a37d-c8729493b79d"). InnerVolumeSpecName "kube-api-access-bq65m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:41.675045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.675008    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-kube-api-access-4lkqp" pod "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" (UID: "d5efe5a3-93a2-4e34-ba6d-1037ac9b0954"). InnerVolumeSpecName "kube-api-access-4lkqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:41.675045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.675039    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93007950-2d7f-495f-a37d-c8729493b79d-istio-token" pod "93007950-2d7f-495f-a37d-c8729493b79d" (UID: "93007950-2d7f-495f-a37d-c8729493b79d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:41.675361 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.675148    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93007950-2d7f-495f-a37d-c8729493b79d-istio-podinfo" pod "93007950-2d7f-495f-a37d-c8729493b79d" (UID: "93007950-2d7f-495f-a37d-c8729493b79d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:46:41.675361 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:41.675283    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d0fc13c282706c183153e71f2b0ac0c3e7c9183b0455f5df2f40aca13d6f89d\": container with ID starting with 1d0fc13c282706c183153e71f2b0ac0c3e7c9183b0455f5df2f40aca13d6f89d not found: ID does not exist" containerID="1d0fc13c282706c183153e71f2b0ac0c3e7c9183b0455f5df2f40aca13d6f89d"
Oct 31 13:46:41.675361 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.675320    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0fc13c282706c183153e71f2b0ac0c3e7c9183b0455f5df2f40aca13d6f89d"} err="failed to get container status \"1d0fc13c282706c183153e71f2b0ac0c3e7c9183b0455f5df2f40aca13d6f89d\": rpc error: code = NotFound desc = could not find container \"1d0fc13c282706c183153e71f2b0ac0c3e7c9183b0455f5df2f40aca13d6f89d\": container with ID starting with 1d0fc13c282706c183153e71f2b0ac0c3e7c9183b0455f5df2f40aca13d6f89d not found: ID does not exist"
Oct 31 13:46:41.675361 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.675347    4428 scope.go:122] "RemoveContainer" containerID="903a102639cbd6fffd6ee7f723cb568f6dc40343b4bc6f4b31fbce3da61165e6"
Oct 31 13:46:41.676407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.676353    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-istio-envoy" pod "93007950-2d7f-495f-a37d-c8729493b79d" (UID: "93007950-2d7f-495f-a37d-c8729493b79d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:46:41.688893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.688835    4428 scope.go:122] "RemoveContainer" containerID="65a8f3089a0bf9247fc8fa15579daf487383d3c6d94707994105b32bc9df7881"
Oct 31 13:46:41.702078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.702052    4428 scope.go:122] "RemoveContainer" containerID="903a102639cbd6fffd6ee7f723cb568f6dc40343b4bc6f4b31fbce3da61165e6"
Oct 31 13:46:41.702437 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:41.702413    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"903a102639cbd6fffd6ee7f723cb568f6dc40343b4bc6f4b31fbce3da61165e6\": container with ID starting with 903a102639cbd6fffd6ee7f723cb568f6dc40343b4bc6f4b31fbce3da61165e6 not found: ID does not exist" containerID="903a102639cbd6fffd6ee7f723cb568f6dc40343b4bc6f4b31fbce3da61165e6"
Oct 31 13:46:41.702520 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.702447    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"903a102639cbd6fffd6ee7f723cb568f6dc40343b4bc6f4b31fbce3da61165e6"} err="failed to get container status \"903a102639cbd6fffd6ee7f723cb568f6dc40343b4bc6f4b31fbce3da61165e6\": rpc error: code = NotFound desc = could not find container \"903a102639cbd6fffd6ee7f723cb568f6dc40343b4bc6f4b31fbce3da61165e6\": container with ID starting with 903a102639cbd6fffd6ee7f723cb568f6dc40343b4bc6f4b31fbce3da61165e6 not found: ID does not exist"
Oct 31 13:46:41.702520 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.702470    4428 scope.go:122] "RemoveContainer" containerID="65a8f3089a0bf9247fc8fa15579daf487383d3c6d94707994105b32bc9df7881"
Oct 31 13:46:41.702744 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:41.702728    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a8f3089a0bf9247fc8fa15579daf487383d3c6d94707994105b32bc9df7881\": container with ID starting with 65a8f3089a0bf9247fc8fa15579daf487383d3c6d94707994105b32bc9df7881 not found: ID does not exist" containerID="65a8f3089a0bf9247fc8fa15579daf487383d3c6d94707994105b32bc9df7881"
Oct 31 13:46:41.702790 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.702749    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a8f3089a0bf9247fc8fa15579daf487383d3c6d94707994105b32bc9df7881"} err="failed to get container status \"65a8f3089a0bf9247fc8fa15579daf487383d3c6d94707994105b32bc9df7881\": rpc error: code = NotFound desc = could not find container \"65a8f3089a0bf9247fc8fa15579daf487383d3c6d94707994105b32bc9df7881\": container with ID starting with 65a8f3089a0bf9247fc8fa15579daf487383d3c6d94707994105b32bc9df7881 not found: ID does not exist"
Oct 31 13:46:41.785899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.781608    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.785899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.781650    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.785899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.781668    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.785899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.781683    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.785899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.781698    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.785899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.781711    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.785899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.781727    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.785899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.781740    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.785899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.781754    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93007950-2d7f-495f-a37d-c8729493b79d-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.785899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.781770    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.785899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.781783    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93007950-2d7f-495f-a37d-c8729493b79d-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.785899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.781797    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq65m\" (UniqueName: \"kubernetes.io/projected/93007950-2d7f-495f-a37d-c8729493b79d-kube-api-access-bq65m\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.785899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.781814    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93007950-2d7f-495f-a37d-c8729493b79d-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.785899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.781828    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lkqp\" (UniqueName: \"kubernetes.io/projected/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-kube-api-access-4lkqp\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.785899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.781841    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.785899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.781855    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.790637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.789962    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.790637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.790015    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93007950-2d7f-495f-a37d-c8729493b79d-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:41.790637 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.788932    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:41.986254 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.986220    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"]
Oct 31 13:46:41.994469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:41.994437    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-bf8c6"]
Oct 31 13:46:42.015429 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:42.015399    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"]
Oct 31 13:46:42.019641 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:42.019613    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-c28ww"]
Oct 31 13:46:42.640363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:42.636344    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:42.758005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:42.757965    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93007950-2d7f-495f-a37d-c8729493b79d" path="/var/lib/kubelet/pods/93007950-2d7f-495f-a37d-c8729493b79d/volumes"
Oct 31 13:46:42.758653 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:42.758629    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" path="/var/lib/kubelet/pods/d5efe5a3-93a2-4e34-ba6d-1037ac9b0954/volumes"
Oct 31 13:46:43.378962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.378927    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"]
Oct 31 13:46:43.379753 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379726    4428 cpu_manager.go:414] "containerMap: removing container" podUID="530cfbed-6986-4f00-9eb5-bb0d7985827e" containerName="payload-processing"
Oct 31 13:46:43.379753 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379744    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530cfbed-6986-4f00-9eb5-bb0d7985827e" containerName="payload-processing"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379769    4428 cpu_manager.go:414] "containerMap: removing container" podUID="5d8cf8de-bfa2-4e2d-8069-e1b1132fe426" containerName="payload-pre-processing"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379777    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d8cf8de-bfa2-4e2d-8069-e1b1132fe426" containerName="payload-pre-processing"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379805    4428 cpu_manager.go:414] "containerMap: removing container" podUID="8c73f7d7-3bac-40e8-b386-3742d5a73225" containerName="payload-pre-processing"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379810    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c73f7d7-3bac-40e8-b386-3742d5a73225" containerName="payload-pre-processing"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379818    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e0b75287-cf60-49fe-8b06-dec5a12b5cd1" containerName="istio-proxy"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379823    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b75287-cf60-49fe-8b06-dec5a12b5cd1" containerName="istio-proxy"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379832    4428 cpu_manager.go:414] "containerMap: removing container" podUID="1fb736b6-9425-445b-85bb-be3f5a9598d1" containerName="payload-pre-processing"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379840    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb736b6-9425-445b-85bb-be3f5a9598d1" containerName="payload-pre-processing"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379850    4428 cpu_manager.go:414] "containerMap: removing container" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379855    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379889    4428 cpu_manager.go:414] "containerMap: removing container" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379895    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379909    4428 cpu_manager.go:414] "containerMap: removing container" podUID="93007950-2d7f-495f-a37d-c8729493b79d" containerName="istio-proxy"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379914    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93007950-2d7f-495f-a37d-c8729493b79d" containerName="istio-proxy"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379922    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" containerName="istio-proxy"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379927    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" containerName="istio-proxy"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379936    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" containerName="istio-proxy"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379941    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" containerName="istio-proxy"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379947    4428 cpu_manager.go:414] "containerMap: removing container" podUID="1e0031a9-bab3-4590-8add-4eeafdf4ac52" containerName="maas-api"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379955    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0031a9-bab3-4590-8add-4eeafdf4ac52" containerName="maas-api"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379968    4428 cpu_manager.go:414] "containerMap: removing container" podUID="6738ac01-586f-4313-9ced-ccb4fbc5cd39" containerName="maas-api"
Oct 31 13:46:43.380041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.379973    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6738ac01-586f-4313-9ced-ccb4fbc5cd39" containerName="maas-api"
Oct 31 13:46:43.382134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.380092    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy"
Oct 31 13:46:43.382134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.380104    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="93007950-2d7f-495f-a37d-c8729493b79d" containerName="istio-proxy"
Oct 31 13:46:43.382134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.380114    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" containerName="istio-proxy"
Oct 31 13:46:43.382134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.380124    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5efe5a3-93a2-4e34-ba6d-1037ac9b0954" containerName="istio-proxy"
Oct 31 13:46:43.382134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.380132    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0031a9-bab3-4590-8add-4eeafdf4ac52" containerName="maas-api"
Oct 31 13:46:43.382134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.380137    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6738ac01-586f-4313-9ced-ccb4fbc5cd39" containerName="maas-api"
Oct 31 13:46:43.382134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.380143    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="530cfbed-6986-4f00-9eb5-bb0d7985827e" containerName="payload-processing"
Oct 31 13:46:43.382134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.380148    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d8cf8de-bfa2-4e2d-8069-e1b1132fe426" containerName="payload-pre-processing"
Oct 31 13:46:43.382134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.380159    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c73f7d7-3bac-40e8-b386-3742d5a73225" containerName="payload-pre-processing"
Oct 31 13:46:43.382134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.380165    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b75287-cf60-49fe-8b06-dec5a12b5cd1" containerName="istio-proxy"
Oct 31 13:46:43.382134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.380171    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb736b6-9425-445b-85bb-be3f5a9598d1" containerName="payload-pre-processing"
Oct 31 13:46:43.386597 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.386567    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:46:43.398926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.393996    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-612b46-serving-cert\""
Oct 31 13:46:43.398926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.398258    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"]
Oct 31 13:46:43.530108 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.526890    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35b067ff-bd37-4cc5-85c9-d877a435751e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7\" (UID: \"35b067ff-bd37-4cc5-85c9-d877a435751e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:46:43.530108 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.526952    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7rhl\" (UniqueName: \"kubernetes.io/projected/35b067ff-bd37-4cc5-85c9-d877a435751e-kube-api-access-l7rhl\") pod \"maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7\" (UID: \"35b067ff-bd37-4cc5-85c9-d877a435751e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:46:43.530108 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.527026    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b067ff-bd37-4cc5-85c9-d877a435751e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7\" (UID: \"35b067ff-bd37-4cc5-85c9-d877a435751e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:46:43.530108 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.527090    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b067ff-bd37-4cc5-85c9-d877a435751e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7\" (UID: \"35b067ff-bd37-4cc5-85c9-d877a435751e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:46:43.536215 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.536147    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:43.629040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.628950    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35b067ff-bd37-4cc5-85c9-d877a435751e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7\" (UID: \"35b067ff-bd37-4cc5-85c9-d877a435751e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:46:43.629040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.628987    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7rhl\" (UniqueName: \"kubernetes.io/projected/35b067ff-bd37-4cc5-85c9-d877a435751e-kube-api-access-l7rhl\") pod \"maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7\" (UID: \"35b067ff-bd37-4cc5-85c9-d877a435751e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:46:43.629040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.629032    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b067ff-bd37-4cc5-85c9-d877a435751e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7\" (UID: \"35b067ff-bd37-4cc5-85c9-d877a435751e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:46:43.629342 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.629068    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b067ff-bd37-4cc5-85c9-d877a435751e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7\" (UID: \"35b067ff-bd37-4cc5-85c9-d877a435751e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:46:43.629826 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.629801    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b067ff-bd37-4cc5-85c9-d877a435751e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7\" (UID: \"35b067ff-bd37-4cc5-85c9-d877a435751e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:46:43.631682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.631658    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35b067ff-bd37-4cc5-85c9-d877a435751e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7\" (UID: \"35b067ff-bd37-4cc5-85c9-d877a435751e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:46:43.631796 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.631729    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b067ff-bd37-4cc5-85c9-d877a435751e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7\" (UID: \"35b067ff-bd37-4cc5-85c9-d877a435751e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:46:43.644264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.644235    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7rhl\" (UniqueName: \"kubernetes.io/projected/35b067ff-bd37-4cc5-85c9-d877a435751e-kube-api-access-l7rhl\") pod \"maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7\" (UID: \"35b067ff-bd37-4cc5-85c9-d877a435751e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:46:43.715463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.715425    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:43.718384 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.718359    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:46:43.878149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:43.878111    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:44.014455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.014421    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"]
Oct 31 13:46:44.025227 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:46:44.025194    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b067ff_bd37_4cc5_85c9_d877a435751e.slice/crio-c2c7072230bc7877a9ffb11f4da1dce2e8d2e19c21245ccac88c0e3f3a4e010c WatchSource:0}: Error finding container c2c7072230bc7877a9ffb11f4da1dce2e8d2e19c21245ccac88c0e3f3a4e010c: Status 404 returned error can't find the container with id c2c7072230bc7877a9ffb11f4da1dce2e8d2e19c21245ccac88c0e3f3a4e010c
Oct 31 13:46:44.596621 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.596593    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:44.703754 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.703705    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7" event={"ID":"35b067ff-bd37-4cc5-85c9-d877a435751e","Type":"ContainerStarted","Data":"4109e0bd0edf309d1628e404b29a16eb685a6f3ac2dba9e030ff07a0af6a27f6"}
Oct 31 13:46:44.703986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.703765    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7" event={"ID":"35b067ff-bd37-4cc5-85c9-d877a435751e","Type":"ContainerStarted","Data":"c2c7072230bc7877a9ffb11f4da1dce2e8d2e19c21245ccac88c0e3f3a4e010c"}
Oct 31 13:46:44.704133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.704104    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:46:44.736702 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.736644    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7" podStartSLOduration=1.195148069 podStartE2EDuration="1.736628146s" totalImagesPullingTime="541.480077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:46:44.731462319 +0000 UTC m=+2602.129810942" watchObservedRunningTime="2026-10-31 13:46:44.736628146 +0000 UTC m=+2602.134976769"
Oct 31 13:46:44.790264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.790228    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:44.832164 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.832086    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"]
Oct 31 13:46:44.833188 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.833161    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a9f039-bd18-4c3a-88a3-973bc6721086" containerName="istio-proxy"
Oct 31 13:46:44.837362 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.837339    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:44.843975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.843949    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-5e8f29\"/\"kube-root-ca.crt\""
Oct 31 13:46:44.844248 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.844227    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-5e8f29\"/\"default-dockercfg-fdkkk\""
Oct 31 13:46:44.844427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.844289    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-5e8f29\"/\"auth-test-model-5e8f29-kserve-self-signed-certs\""
Oct 31 13:46:44.845224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.844431    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-5e8f29\"/\"openshift-service-ca.crt\""
Oct 31 13:46:44.850611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.850582    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"]
Oct 31 13:46:44.961933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.961896    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-model-cache\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:44.961933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.961930    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a034fcda-0a8c-44b7-ab07-4e36f9520135-tls-certs\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:44.962247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.961962    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-tmp-dir\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:44.962247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.962205    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-dshm\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:44.962331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.962315    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzchx\" (UniqueName: \"kubernetes.io/projected/a034fcda-0a8c-44b7-ab07-4e36f9520135-kube-api-access-zzchx\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:44.962381 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.962369    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-home\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:44.967451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:44.967427    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:45.068988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:45.068949    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-dshm\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:45.069228 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:45.069123    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzchx\" (UniqueName: \"kubernetes.io/projected/a034fcda-0a8c-44b7-ab07-4e36f9520135-kube-api-access-zzchx\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:45.069557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:45.069521    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-home\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:45.069686 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:45.069665    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-model-cache\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:45.074099 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:45.074075    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a034fcda-0a8c-44b7-ab07-4e36f9520135-tls-certs\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:45.074779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:45.074757    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-tmp-dir\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:45.075582 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:45.069974    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-home\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:45.075949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:45.075194    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-tmp-dir\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:45.088993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:45.074021    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-model-cache\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:45.089779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:45.089752    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a034fcda-0a8c-44b7-ab07-4e36f9520135-tls-certs\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:45.098816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:45.098754    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzchx\" (UniqueName: \"kubernetes.io/projected/a034fcda-0a8c-44b7-ab07-4e36f9520135-kube-api-access-zzchx\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:45.117184 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:45.117116    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-dshm\") pod \"auth-test-model-5e8f29-kserve-5ccf44b869-vrp68\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") " pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:45.170776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:45.170481    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:45.641912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:45.641788    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:45.765140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:45.765107    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"]
Oct 31 13:46:45.776859 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:46:45.776828    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda034fcda_0a8c_44b7_ab07_4e36f9520135.slice/crio-84764ef6663fc38dc7c2d370f25aef0fc0dd8d13817b5b97e9043434e017e85f WatchSource:0}: Error finding container 84764ef6663fc38dc7c2d370f25aef0fc0dd8d13817b5b97e9043434e017e85f: Status 404 returned error can't find the container with id 84764ef6663fc38dc7c2d370f25aef0fc0dd8d13817b5b97e9043434e017e85f
Oct 31 13:46:46.523976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:46.523931    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:46.684393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:46.684361    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:46.782365 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:46.781683    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68" event={"ID":"a034fcda-0a8c-44b7-ab07-4e36f9520135","Type":"ContainerStarted","Data":"c6d0b3ec135567f265530384039ce81ad7eaad2899bc5a3abeabb086fb95b963"}
Oct 31 13:46:46.782634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:46.782600    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68" event={"ID":"a034fcda-0a8c-44b7-ab07-4e36f9520135","Type":"ContainerStarted","Data":"84764ef6663fc38dc7c2d370f25aef0fc0dd8d13817b5b97e9043434e017e85f"}
Oct 31 13:46:46.810066 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:46.809995    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68" podStartSLOduration=2.8099750390000002 podStartE2EDuration="2.809975039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:46:46.803533209 +0000 UTC m=+2604.201881833" watchObservedRunningTime="2026-10-31 13:46:46.809975039 +0000 UTC m=+2604.208323665"
Oct 31 13:46:46.867432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:46.867396    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:47.035617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:47.035426    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:47.181910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:47.179976    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:47.667894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:47.666366    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:47.848247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:47.848215    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:48.165938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.165898    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"]
Oct 31 13:46:48.171057 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.171029    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.178124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.178099    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-97927b-openshift-default-dockercfg-p88tk\""
Oct 31 13:46:48.179617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.179582    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"]
Oct 31 13:46:48.184659 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.184630    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-workload-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.184801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.184709    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-podinfo\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.184975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.184947    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-envoy\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.185055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.184996    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-data\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.185055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.185030    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-token\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.185180 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.185064    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-workload-certs\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.185180 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.185098    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istiod-ca-cert\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.185180 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.185140    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k2fb\" (UniqueName: \"kubernetes.io/projected/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-kube-api-access-9k2fb\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.185308 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.185195    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-credential-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.265147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.265109    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:48.312904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.305146    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-workload-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.312904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.305242    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-podinfo\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.312904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.305396    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-envoy\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.312904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.305427    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-data\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.312904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.305453    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-token\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.312904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.305484    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-workload-certs\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.312904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.305516    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istiod-ca-cert\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.312904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.305554    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k2fb\" (UniqueName: \"kubernetes.io/projected/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-kube-api-access-9k2fb\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.312904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.305610    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-credential-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.335893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.313975    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-credential-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.336175 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.321350    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-workload-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.336297 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.333226    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-workload-certs\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.336394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.333853    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istiod-ca-cert\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.336486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.334499    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-data\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.357896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.355226    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-envoy\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.399905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.397188    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-podinfo\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.421938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.419543    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k2fb\" (UniqueName: \"kubernetes.io/projected/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-kube-api-access-9k2fb\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.438895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.436513    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-token\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.455895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.447845    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:48.492899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.490151    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:48.743904 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:48.743826    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:46:48.838520 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.838489    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:48.868208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.868175    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"]
Oct 31 13:46:48.868666 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:46:48.868643    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7c955c_ca35_4c0d_8625_b06d7b2bf73e.slice/crio-6810a13671a376dcc5a6a333980ee58dcee333da8aa8dea41e4c5ad8d7a9d0e0 WatchSource:0}: Error finding container 6810a13671a376dcc5a6a333980ee58dcee333da8aa8dea41e4c5ad8d7a9d0e0: Status 404 returned error can't find the container with id 6810a13671a376dcc5a6a333980ee58dcee333da8aa8dea41e4c5ad8d7a9d0e0
Oct 31 13:46:48.871694 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.871656    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:46:48.871797 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.871730    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:46:48.871797 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:48.871761    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:46:49.743692 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:49.743560    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:49.838084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:49.838050    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5" event={"ID":"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e","Type":"ContainerStarted","Data":"694883a8256ff938e1e8e8358a22a3d5b7b1ed9c6480a4188447472456c64804"}
Oct 31 13:46:49.838320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:49.838115    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5" event={"ID":"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e","Type":"ContainerStarted","Data":"6810a13671a376dcc5a6a333980ee58dcee333da8aa8dea41e4c5ad8d7a9d0e0"}
Oct 31 13:46:49.874665 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:49.874602    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5" podStartSLOduration=1.874579792 podStartE2EDuration="1.874579792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:46:49.870861176 +0000 UTC m=+2607.269209800" watchObservedRunningTime="2026-10-31 13:46:49.874579792 +0000 UTC m=+2607.272928416"
Oct 31 13:46:50.491144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:50.491065    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:50.499062 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:50.499030    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:50.646181 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:50.644391    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:50.757077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:50.756989    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:46:50.851505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:50.851464    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:50.852635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:50.852605    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:46:51.801891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:51.797036    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:52.231386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:52.231346    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:52.415985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:52.415898    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:53.209314 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:53.209276    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:53.540100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:53.540057    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:53.728421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:53.728385    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:54.298701 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:54.298641    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:54.878069 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:54.878027    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"]
Oct 31 13:46:54.878311 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:54.878286    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk" podUID="545cc90a-0eda-4923-9757-c351ccac5d73" containerName="maas-api" containerID="cri-o://b66d28ec0ae214f95041df445e37e1658eecd2ed5974576181af74076a3f69ac" gracePeriod=30
Oct 31 13:46:54.991067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:54.991033    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:55.346809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.324960    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:55.349782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.349737    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:55.379544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.379511    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:55.419258 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.419233    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:46:55.509076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.509042    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:55.517391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.517365    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/545cc90a-0eda-4923-9757-c351ccac5d73-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/545cc90a-0eda-4923-9757-c351ccac5d73-openshift-service-ca\") pod \"545cc90a-0eda-4923-9757-c351ccac5d73\" (UID: \"545cc90a-0eda-4923-9757-c351ccac5d73\") "
Oct 31 13:46:55.517520 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.517413    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/545cc90a-0eda-4923-9757-c351ccac5d73-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/545cc90a-0eda-4923-9757-c351ccac5d73-maas-api-tls\") pod \"545cc90a-0eda-4923-9757-c351ccac5d73\" (UID: \"545cc90a-0eda-4923-9757-c351ccac5d73\") "
Oct 31 13:46:55.517520 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.517439    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/545cc90a-0eda-4923-9757-c351ccac5d73-metrics-certs\" (UniqueName: \"kubernetes.io/secret/545cc90a-0eda-4923-9757-c351ccac5d73-metrics-certs\") pod \"545cc90a-0eda-4923-9757-c351ccac5d73\" (UID: \"545cc90a-0eda-4923-9757-c351ccac5d73\") "
Oct 31 13:46:55.517599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.517581    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/545cc90a-0eda-4923-9757-c351ccac5d73-kube-api-access-vf7pz\" (UniqueName: \"kubernetes.io/projected/545cc90a-0eda-4923-9757-c351ccac5d73-kube-api-access-vf7pz\") pod \"545cc90a-0eda-4923-9757-c351ccac5d73\" (UID: \"545cc90a-0eda-4923-9757-c351ccac5d73\") "
Oct 31 13:46:55.517778 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.517754    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/545cc90a-0eda-4923-9757-c351ccac5d73-openshift-service-ca" pod "545cc90a-0eda-4923-9757-c351ccac5d73" (UID: "545cc90a-0eda-4923-9757-c351ccac5d73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:46:55.518369 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.518342    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/545cc90a-0eda-4923-9757-c351ccac5d73-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:55.519952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.519925    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/545cc90a-0eda-4923-9757-c351ccac5d73-maas-api-tls" pod "545cc90a-0eda-4923-9757-c351ccac5d73" (UID: "545cc90a-0eda-4923-9757-c351ccac5d73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:46:55.519952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.519934    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545cc90a-0eda-4923-9757-c351ccac5d73-kube-api-access-vf7pz" pod "545cc90a-0eda-4923-9757-c351ccac5d73" (UID: "545cc90a-0eda-4923-9757-c351ccac5d73"). InnerVolumeSpecName "kube-api-access-vf7pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:46:55.520095 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.520001    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/545cc90a-0eda-4923-9757-c351ccac5d73-metrics-certs" pod "545cc90a-0eda-4923-9757-c351ccac5d73" (UID: "545cc90a-0eda-4923-9757-c351ccac5d73"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:46:55.622839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.622459    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/545cc90a-0eda-4923-9757-c351ccac5d73-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:55.623099 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.623079    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/545cc90a-0eda-4923-9757-c351ccac5d73-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:55.623202 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.623189    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf7pz\" (UniqueName: \"kubernetes.io/projected/545cc90a-0eda-4923-9757-c351ccac5d73-kube-api-access-vf7pz\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:46:55.912070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.911970    4428 generic.go:356] "Generic (PLEG): container finished" podID="545cc90a-0eda-4923-9757-c351ccac5d73" containerID="b66d28ec0ae214f95041df445e37e1658eecd2ed5974576181af74076a3f69ac" exitCode=0
Oct 31 13:46:55.912070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.912030    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"
Oct 31 13:46:55.912070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.912052    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk" event={"ID":"545cc90a-0eda-4923-9757-c351ccac5d73","Type":"ContainerDied","Data":"b66d28ec0ae214f95041df445e37e1658eecd2ed5974576181af74076a3f69ac"}
Oct 31 13:46:55.912371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.912115    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk" event={"ID":"545cc90a-0eda-4923-9757-c351ccac5d73","Type":"ContainerDied","Data":"e547f1394423797351e49bc90460b80f8deac6c54d924cdf2be0d1fe60b9ed31"}
Oct 31 13:46:55.912371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.912138    4428 scope.go:122] "RemoveContainer" containerID="b66d28ec0ae214f95041df445e37e1658eecd2ed5974576181af74076a3f69ac"
Oct 31 13:46:55.929641 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.929612    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:46:55.930972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.930954    4428 scope.go:122] "RemoveContainer" containerID="b66d28ec0ae214f95041df445e37e1658eecd2ed5974576181af74076a3f69ac"
Oct 31 13:46:55.931317 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:46:55.931294    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b66d28ec0ae214f95041df445e37e1658eecd2ed5974576181af74076a3f69ac\": container with ID starting with b66d28ec0ae214f95041df445e37e1658eecd2ed5974576181af74076a3f69ac not found: ID does not exist" containerID="b66d28ec0ae214f95041df445e37e1658eecd2ed5974576181af74076a3f69ac"
Oct 31 13:46:55.931377 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.931330    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b66d28ec0ae214f95041df445e37e1658eecd2ed5974576181af74076a3f69ac"} err="failed to get container status \"b66d28ec0ae214f95041df445e37e1658eecd2ed5974576181af74076a3f69ac\": rpc error: code = NotFound desc = could not find container \"b66d28ec0ae214f95041df445e37e1658eecd2ed5974576181af74076a3f69ac\": container with ID starting with b66d28ec0ae214f95041df445e37e1658eecd2ed5974576181af74076a3f69ac not found: ID does not exist"
Oct 31 13:46:55.936154 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.936129    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:55.944814 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.944787    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"]
Oct 31 13:46:55.948983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:55.948961    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cae9ec-6567c4797c-7jzhk"]
Oct 31 13:46:56.600902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:56.595923    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:56.757936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:56.757902    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="545cc90a-0eda-4923-9757-c351ccac5d73" path="/var/lib/kubelet/pods/545cc90a-0eda-4923-9757-c351ccac5d73/volumes"
Oct 31 13:46:57.564893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:57.562967    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:58.657144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:58.657105    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:46:59.434895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:46:59.431423    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:00.061534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:00.061486    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:00.305007 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:00.303527    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:00.682171 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:00.682137    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:00.945367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:00.945279    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:01.218849 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:01.218815    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:02.181487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:02.155010    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:03.190898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:03.180816    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:04.136893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:04.128308    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:04.797896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:04.797716    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:04.963663 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:04.963634    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:05.035115 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:05.035051    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"]
Oct 31 13:47:05.036200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:05.036113    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerName="istio-proxy" containerID="cri-o://568dc5b3f98f4a9f69d2e9e48db34228213d897cf4eac287edc5071280cc1340" gracePeriod=30
Oct 31 13:47:05.145780 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:05.145669    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:05.317834 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:05.317760    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:05.915670 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:05.915637    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:06.192544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.192468    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:06.246558 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.246520    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"]
Oct 31 13:47:06.247755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.247727    4428 cpu_manager.go:414] "containerMap: removing container" podUID="545cc90a-0eda-4923-9757-c351ccac5d73" containerName="maas-api"
Oct 31 13:47:06.247755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.247755    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="545cc90a-0eda-4923-9757-c351ccac5d73" containerName="maas-api"
Oct 31 13:47:06.248055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.248036    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="545cc90a-0eda-4923-9757-c351ccac5d73" containerName="maas-api"
Oct 31 13:47:06.254414 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.254387    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.259938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.259909    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-a39e7e-openshift-default-dockercfg-x278n\""
Oct 31 13:47:06.263128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.263098    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"]
Oct 31 13:47:06.420711 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.420661    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:06.431910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.426292    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-workload-certs\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.431910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.426350    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-podinfo\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.431910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.426389    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnlbd\" (UniqueName: \"kubernetes.io/projected/5701fca5-38a3-4a28-86b3-55124cb6f54e-kube-api-access-cnlbd\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.431910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.426471    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-workload-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.431910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.426581    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-token\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.431910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.426610    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5701fca5-38a3-4a28-86b3-55124cb6f54e-istiod-ca-cert\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.431910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.426763    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-envoy\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.431910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.427009    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-credential-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.431910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.427139    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-data\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.542039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.536459    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-envoy\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.542039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.536553    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-credential-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.542039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.536718    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-data\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.542039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.536807    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-workload-certs\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.542039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.536860    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-podinfo\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.542039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.536969    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnlbd\" (UniqueName: \"kubernetes.io/projected/5701fca5-38a3-4a28-86b3-55124cb6f54e-kube-api-access-cnlbd\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.542039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.537064    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-workload-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.542039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.537172    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-token\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.542039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.537201    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5701fca5-38a3-4a28-86b3-55124cb6f54e-istiod-ca-cert\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.550442 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.550390    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5701fca5-38a3-4a28-86b3-55124cb6f54e-istiod-ca-cert\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.557935 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.557433    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-workload-certs\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.593639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.576614    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-envoy\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.593639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.585916    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-workload-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.593639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.587433    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-data\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.593639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.589049    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-credential-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.595827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.595792    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:06.629891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.626657    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnlbd\" (UniqueName: \"kubernetes.io/projected/5701fca5-38a3-4a28-86b3-55124cb6f54e-kube-api-access-cnlbd\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.643633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.640265    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-podinfo\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.704041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.703990    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-token\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:06.864700 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.864594    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:06.880101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:06.880066    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:07.180077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:07.180047    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"]
Oct 31 13:47:07.180415 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:47:07.180394    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5701fca5_38a3_4a28_86b3_55124cb6f54e.slice/crio-8f0c38cefc18d6936ee1c6f80ad8c028fc03c4c6f2338f59dd2fa76de783110e WatchSource:0}: Error finding container 8f0c38cefc18d6936ee1c6f80ad8c028fc03c4c6f2338f59dd2fa76de783110e: Status 404 returned error can't find the container with id 8f0c38cefc18d6936ee1c6f80ad8c028fc03c4c6f2338f59dd2fa76de783110e
Oct 31 13:47:07.183295 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:07.183261    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:47:07.183360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:07.183333    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:47:07.183406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:07.183362    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:47:07.554910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:07.553461    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:08.064758 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:08.063957    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh" event={"ID":"5701fca5-38a3-4a28-86b3-55124cb6f54e","Type":"ContainerStarted","Data":"efe28536f4ea94d65dbf1603e4354ba9c1982b453fc165400e6ca54cec638912"}
Oct 31 13:47:08.064758 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:08.064040    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh" event={"ID":"5701fca5-38a3-4a28-86b3-55124cb6f54e","Type":"ContainerStarted","Data":"8f0c38cefc18d6936ee1c6f80ad8c028fc03c4c6f2338f59dd2fa76de783110e"}
Oct 31 13:47:08.106091 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:08.106021    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh" podStartSLOduration=2.106002292 podStartE2EDuration="2.106002292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:08.105988893 +0000 UTC m=+2625.504337519" watchObservedRunningTime="2026-10-31 13:47:08.106002292 +0000 UTC m=+2625.504350916"
Oct 31 13:47:08.816362 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:08.813404    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:08.880727 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:08.880693    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:08.887337 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:08.887310    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:08.997676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:08.997413    4428 patch_prober.go:29] interesting pod/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:47:08.997676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:08.997498    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:47:09.007321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.007287    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:09.072575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.072492    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:09.073537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.073517    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:47:09.225024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.224989    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"]
Oct 31 13:47:09.225324 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.225246    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp" podUID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" containerName="istio-proxy" containerID="cri-o://4ad106c9ce07fd90a1994a884773eb13ec19a77c5fa44fb8675dd543be7c005e" gracePeriod=30
Oct 31 13:47:09.307554 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.307430    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:09.565941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.563270    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"]
Oct 31 13:47:09.568780 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.568755    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.569555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.569524    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"]
Oct 31 13:47:09.587359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.587328    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-6a3533\"/\"default-dockercfg-x8ws9\""
Oct 31 13:47:09.587619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.587590    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-6a3533\"/\"auth-test-model-6a3533-kserve-self-signed-certs\""
Oct 31 13:47:09.589143 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.589115    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-6a3533\"/\"openshift-service-ca.crt\""
Oct 31 13:47:09.589271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.589244    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-6a3533\"/\"kube-root-ca.crt\""
Oct 31 13:47:09.663319 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.663284    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:09.679507 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.679472    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-dshm\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.679699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.679517    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-tls-certs\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.679744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.679693    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-model-cache\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.679956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.679938    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-tmp-dir\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.680022 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.680002    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4mz9\" (UniqueName: \"kubernetes.io/projected/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-kube-api-access-d4mz9\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.680086 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.680068    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-home\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.781468 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.781431    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-tmp-dir\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.781671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.781494    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4mz9\" (UniqueName: \"kubernetes.io/projected/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-kube-api-access-d4mz9\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.781671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.781516    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-home\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.781671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.781550    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-dshm\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.781671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.781569    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-tls-certs\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.781671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.781659    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-model-cache\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.781975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.781924    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-tmp-dir\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.782016 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.781976    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-home\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.782136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.782110    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-model-cache\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.784206 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.784182    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-dshm\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.784405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.784387    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-tls-certs\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.820940 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.798188    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4mz9\" (UniqueName: \"kubernetes.io/projected/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-kube-api-access-d4mz9\") pod \"auth-test-model-6a3533-kserve-679c56786f-rp8b4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") " pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:09.896961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:09.895656    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:10.108430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.108389    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:10.265900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.265839    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"]
Oct 31 13:47:10.270053 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:47:10.270014    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38ef9813_0afd_477c_8e9f_1bcbfbfd17a4.slice/crio-9ea6937c19cd76e735357bc272b018577d344593a13c75195c50195cceabaee6 WatchSource:0}: Error finding container 9ea6937c19cd76e735357bc272b018577d344593a13c75195c50195cceabaee6: Status 404 returned error can't find the container with id 9ea6937c19cd76e735357bc272b018577d344593a13c75195c50195cceabaee6
Oct 31 13:47:10.338945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.318256    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:10.453096 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.453059    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:47:10.544514 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.544480    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-podinfo\") pod \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") "
Oct 31 13:47:10.544690 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.544556    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-workload-certs\") pod \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") "
Oct 31 13:47:10.544690 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.544616    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istiod-ca-cert\") pod \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") "
Oct 31 13:47:10.544690 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.544674    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-data\") pod \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") "
Oct 31 13:47:10.544862 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.544783    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-workload-socket\") pod \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") "
Oct 31 13:47:10.544862 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.544815    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-envoy\") pod \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") "
Oct 31 13:47:10.544862 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.544849    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-credential-socket\") pod \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") "
Oct 31 13:47:10.545039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.544908    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-token\" (UniqueName: \"kubernetes.io/projected/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-token\") pod \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") "
Oct 31 13:47:10.545039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.544988    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-kube-api-access-9sbsx\" (UniqueName: \"kubernetes.io/projected/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-kube-api-access-9sbsx\") pod \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\" (UID: \"d1af67f4-a0eb-402b-8453-7cb5ab463ab1\") "
Oct 31 13:47:10.546339 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.546220    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-data" pod "d1af67f4-a0eb-402b-8453-7cb5ab463ab1" (UID: "d1af67f4-a0eb-402b-8453-7cb5ab463ab1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:47:10.547735 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.547692    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:10.548360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.548335    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istiod-ca-cert" pod "d1af67f4-a0eb-402b-8453-7cb5ab463ab1" (UID: "d1af67f4-a0eb-402b-8453-7cb5ab463ab1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:47:10.548600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.548568    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-credential-socket" pod "d1af67f4-a0eb-402b-8453-7cb5ab463ab1" (UID: "d1af67f4-a0eb-402b-8453-7cb5ab463ab1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:47:10.548700 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.548558    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-workload-certs" pod "d1af67f4-a0eb-402b-8453-7cb5ab463ab1" (UID: "d1af67f4-a0eb-402b-8453-7cb5ab463ab1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:47:10.549350 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.549314    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-workload-socket" pod "d1af67f4-a0eb-402b-8453-7cb5ab463ab1" (UID: "d1af67f4-a0eb-402b-8453-7cb5ab463ab1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:47:10.549828 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.549803    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-kube-api-access-9sbsx" pod "d1af67f4-a0eb-402b-8453-7cb5ab463ab1" (UID: "d1af67f4-a0eb-402b-8453-7cb5ab463ab1"). InnerVolumeSpecName "kube-api-access-9sbsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:47:10.551517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.551490    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-token" pod "d1af67f4-a0eb-402b-8453-7cb5ab463ab1" (UID: "d1af67f4-a0eb-402b-8453-7cb5ab463ab1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:47:10.552347 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.552132    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-envoy" pod "d1af67f4-a0eb-402b-8453-7cb5ab463ab1" (UID: "d1af67f4-a0eb-402b-8453-7cb5ab463ab1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:47:10.561214 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.561106    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-podinfo" pod "d1af67f4-a0eb-402b-8453-7cb5ab463ab1" (UID: "d1af67f4-a0eb-402b-8453-7cb5ab463ab1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:47:10.670968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.665315    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:10.670968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.665358    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:10.670968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.665373    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:10.670968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.665387    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:10.670968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.665403    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:10.670968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.665416    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:10.670968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.665430    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:10.670968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.665444    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sbsx\" (UniqueName: \"kubernetes.io/projected/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-kube-api-access-9sbsx\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:10.670968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.665461    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1af67f4-a0eb-402b-8453-7cb5ab463ab1-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:10.740648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.740580    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:10.923811 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:10.923715    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:11.093150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.093108    4428 generic.go:356] "Generic (PLEG): container finished" podID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerID="568dc5b3f98f4a9f69d2e9e48db34228213d897cf4eac287edc5071280cc1340" exitCode=0
Oct 31 13:47:11.093344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.093181    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"
Oct 31 13:47:11.093344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.093180    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" event={"ID":"d1af67f4-a0eb-402b-8453-7cb5ab463ab1","Type":"ContainerDied","Data":"568dc5b3f98f4a9f69d2e9e48db34228213d897cf4eac287edc5071280cc1340"}
Oct 31 13:47:11.093344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.093334    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f" event={"ID":"d1af67f4-a0eb-402b-8453-7cb5ab463ab1","Type":"ContainerDied","Data":"8852e6431856e545e349c6fd32ebf3c69f91057f7ed4b6d9cc67ab643c7321c6"}
Oct 31 13:47:11.093505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.093363    4428 scope.go:122] "RemoveContainer" containerID="568dc5b3f98f4a9f69d2e9e48db34228213d897cf4eac287edc5071280cc1340"
Oct 31 13:47:11.095235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.095204    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4" event={"ID":"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4","Type":"ContainerStarted","Data":"f430bea93637a6d60cb8fef4b705aa7fe0531b7b43e35d8ceb5f07e22258b808"}
Oct 31 13:47:11.095391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.095248    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4" event={"ID":"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4","Type":"ContainerStarted","Data":"9ea6937c19cd76e735357bc272b018577d344593a13c75195c50195cceabaee6"}
Oct 31 13:47:11.105654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.105633    4428 scope.go:122] "RemoveContainer" containerID="9fb715f96f38b48403dcf2b346071307d3e228d24b0cac0cc53101e386d544c5"
Oct 31 13:47:11.117295 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.117269    4428 scope.go:122] "RemoveContainer" containerID="568dc5b3f98f4a9f69d2e9e48db34228213d897cf4eac287edc5071280cc1340"
Oct 31 13:47:11.117613 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:47:11.117592    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"568dc5b3f98f4a9f69d2e9e48db34228213d897cf4eac287edc5071280cc1340\": container with ID starting with 568dc5b3f98f4a9f69d2e9e48db34228213d897cf4eac287edc5071280cc1340 not found: ID does not exist" containerID="568dc5b3f98f4a9f69d2e9e48db34228213d897cf4eac287edc5071280cc1340"
Oct 31 13:47:11.117697 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.117621    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"568dc5b3f98f4a9f69d2e9e48db34228213d897cf4eac287edc5071280cc1340"} err="failed to get container status \"568dc5b3f98f4a9f69d2e9e48db34228213d897cf4eac287edc5071280cc1340\": rpc error: code = NotFound desc = could not find container \"568dc5b3f98f4a9f69d2e9e48db34228213d897cf4eac287edc5071280cc1340\": container with ID starting with 568dc5b3f98f4a9f69d2e9e48db34228213d897cf4eac287edc5071280cc1340 not found: ID does not exist"
Oct 31 13:47:11.117697 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.117639    4428 scope.go:122] "RemoveContainer" containerID="9fb715f96f38b48403dcf2b346071307d3e228d24b0cac0cc53101e386d544c5"
Oct 31 13:47:11.117969 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:47:11.117947    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb715f96f38b48403dcf2b346071307d3e228d24b0cac0cc53101e386d544c5\": container with ID starting with 9fb715f96f38b48403dcf2b346071307d3e228d24b0cac0cc53101e386d544c5 not found: ID does not exist" containerID="9fb715f96f38b48403dcf2b346071307d3e228d24b0cac0cc53101e386d544c5"
Oct 31 13:47:11.118024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.117975    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb715f96f38b48403dcf2b346071307d3e228d24b0cac0cc53101e386d544c5"} err="failed to get container status \"9fb715f96f38b48403dcf2b346071307d3e228d24b0cac0cc53101e386d544c5\": rpc error: code = NotFound desc = could not find container \"9fb715f96f38b48403dcf2b346071307d3e228d24b0cac0cc53101e386d544c5\": container with ID starting with 9fb715f96f38b48403dcf2b346071307d3e228d24b0cac0cc53101e386d544c5 not found: ID does not exist"
Oct 31 13:47:11.121438 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.121413    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"]
Oct 31 13:47:11.127078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.127057    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-g2c5f"]
Oct 31 13:47:11.142245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.142200    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4" podStartSLOduration=2.142186606 podStartE2EDuration="2.142186606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:11.139964312 +0000 UTC m=+2628.538312937" watchObservedRunningTime="2026-10-31 13:47:11.142186606 +0000 UTC m=+2628.540535253"
Oct 31 13:47:11.863827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.863793    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:11.909051 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.909010    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:11.927302 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.927265    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"]
Oct 31 13:47:11.928492 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.928464    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerName="istio-proxy"
Oct 31 13:47:11.928492 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.928488    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerName="istio-proxy"
Oct 31 13:47:11.928685 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.928530    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerName="istio-proxy"
Oct 31 13:47:11.928685 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.928553    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerName="istio-proxy"
Oct 31 13:47:11.928768 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.928753    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerName="istio-proxy"
Oct 31 13:47:11.935109 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.935079    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"
Oct 31 13:47:11.942576 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.942499    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-a39e7e-dockercfg-qdtds\""
Oct 31 13:47:11.947962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:11.947509    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"]
Oct 31 13:47:12.011029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:12.010743    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d678e30-767d-48e8-8c7a-74c31e08ed30-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726\" (UID: \"2d678e30-767d-48e8-8c7a-74c31e08ed30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"
Oct 31 13:47:12.011029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:12.010801    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbrd6\" (UniqueName: \"kubernetes.io/projected/2d678e30-767d-48e8-8c7a-74c31e08ed30-kube-api-access-cbrd6\") pod \"payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726\" (UID: \"2d678e30-767d-48e8-8c7a-74c31e08ed30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"
Oct 31 13:47:12.076969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:12.070183    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-a39e7e\""
Oct 31 13:47:12.141949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:12.136256    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d678e30-767d-48e8-8c7a-74c31e08ed30-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726\" (UID: \"2d678e30-767d-48e8-8c7a-74c31e08ed30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"
Oct 31 13:47:12.141949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:12.136327    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbrd6\" (UniqueName: \"kubernetes.io/projected/2d678e30-767d-48e8-8c7a-74c31e08ed30-kube-api-access-cbrd6\") pod \"payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726\" (UID: \"2d678e30-767d-48e8-8c7a-74c31e08ed30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"
Oct 31 13:47:12.141949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:12.137630    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d678e30-767d-48e8-8c7a-74c31e08ed30-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726\" (UID: \"2d678e30-767d-48e8-8c7a-74c31e08ed30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"
Oct 31 13:47:12.141949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:12.141157    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:12.289031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:12.288994    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbrd6\" (UniqueName: \"kubernetes.io/projected/2d678e30-767d-48e8-8c7a-74c31e08ed30-kube-api-access-cbrd6\") pod \"payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726\" (UID: \"2d678e30-767d-48e8-8c7a-74c31e08ed30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"
Oct 31 13:47:12.289743 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:12.289717    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"
Oct 31 13:47:12.593281 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:47:12.593250    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d678e30_767d_48e8_8c7a_74c31e08ed30.slice/crio-c21ef0776925caa5f952e9efcabb219ef732221d4e368a4a30e072d8ddf64796 WatchSource:0}: Error finding container c21ef0776925caa5f952e9efcabb219ef732221d4e368a4a30e072d8ddf64796: Status 404 returned error can't find the container with id c21ef0776925caa5f952e9efcabb219ef732221d4e368a4a30e072d8ddf64796
Oct 31 13:47:12.594015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:12.593822    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"]
Oct 31 13:47:12.766373 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:12.766337    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" path="/var/lib/kubelet/pods/d1af67f4-a0eb-402b-8453-7cb5ab463ab1/volumes"
Oct 31 13:47:13.010387 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:13.010349    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:13.322521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:13.322408    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726" event={"ID":"2d678e30-767d-48e8-8c7a-74c31e08ed30","Type":"ContainerStarted","Data":"2fc06fc19f62370a16d79829d3d20278875294a17f6059de69c29bf6eb599181"}
Oct 31 13:47:13.322521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:13.322479    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726" event={"ID":"2d678e30-767d-48e8-8c7a-74c31e08ed30","Type":"ContainerStarted","Data":"c21ef0776925caa5f952e9efcabb219ef732221d4e368a4a30e072d8ddf64796"}
Oct 31 13:47:13.344378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:13.344324    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726" podStartSLOduration=2.344309179 podStartE2EDuration="2.344309179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:13.34419628 +0000 UTC m=+2630.742544904" watchObservedRunningTime="2026-10-31 13:47:13.344309179 +0000 UTC m=+2630.742657803"
Oct 31 13:47:13.640811 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:13.640724    4428 patch_prober.go:29] interesting pod/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:47:13.640811 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:13.640788    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp" podUID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:47:14.216211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.216146    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:14.359777 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.359742    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"
Oct 31 13:47:14.413343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.413312    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:14.575744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.575694    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:14.643307 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.643276    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:47:14.788751 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.788658    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-token\" (UniqueName: \"kubernetes.io/projected/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-token\") pod \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") "
Oct 31 13:47:14.788751 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.788723    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-credential-socket\") pod \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") "
Oct 31 13:47:14.789062 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.788781    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istiod-ca-cert\") pod \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") "
Oct 31 13:47:14.789062 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.788808    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-envoy\") pod \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") "
Oct 31 13:47:14.789062 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.789005    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-workload-socket\") pod \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") "
Oct 31 13:47:14.789062 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.789060    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-workload-certs\") pod \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") "
Oct 31 13:47:14.789259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.789127    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-data\") pod \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") "
Oct 31 13:47:14.789259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.789156    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-credential-socket" pod "ace3362d-9bb7-46e2-b218-4e7d3e0ed354" (UID: "ace3362d-9bb7-46e2-b218-4e7d3e0ed354"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:47:14.789259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.789246    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-kube-api-access-lx66n\" (UniqueName: \"kubernetes.io/projected/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-kube-api-access-lx66n\") pod \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") "
Oct 31 13:47:14.789434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.789290    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-workload-socket" pod "ace3362d-9bb7-46e2-b218-4e7d3e0ed354" (UID: "ace3362d-9bb7-46e2-b218-4e7d3e0ed354"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:47:14.789434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.789315    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-podinfo\") pod \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\" (UID: \"ace3362d-9bb7-46e2-b218-4e7d3e0ed354\") "
Oct 31 13:47:14.789434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.789346    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istiod-ca-cert" pod "ace3362d-9bb7-46e2-b218-4e7d3e0ed354" (UID: "ace3362d-9bb7-46e2-b218-4e7d3e0ed354"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:47:14.789575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.789436    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-workload-certs" pod "ace3362d-9bb7-46e2-b218-4e7d3e0ed354" (UID: "ace3362d-9bb7-46e2-b218-4e7d3e0ed354"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:47:14.789575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.789469    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-data" pod "ace3362d-9bb7-46e2-b218-4e7d3e0ed354" (UID: "ace3362d-9bb7-46e2-b218-4e7d3e0ed354"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:47:14.790027 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.789997    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:14.790027 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.790026    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:14.790240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.790036    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:14.790240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.790045    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:14.790240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.790054    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:14.791307 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.791283    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-token" pod "ace3362d-9bb7-46e2-b218-4e7d3e0ed354" (UID: "ace3362d-9bb7-46e2-b218-4e7d3e0ed354"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:47:14.791413 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.791336    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-envoy" pod "ace3362d-9bb7-46e2-b218-4e7d3e0ed354" (UID: "ace3362d-9bb7-46e2-b218-4e7d3e0ed354"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:47:14.792129 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.792107    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-podinfo" pod "ace3362d-9bb7-46e2-b218-4e7d3e0ed354" (UID: "ace3362d-9bb7-46e2-b218-4e7d3e0ed354"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:47:14.792129 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.792121    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-kube-api-access-lx66n" pod "ace3362d-9bb7-46e2-b218-4e7d3e0ed354" (UID: "ace3362d-9bb7-46e2-b218-4e7d3e0ed354"). InnerVolumeSpecName "kube-api-access-lx66n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:47:14.891632 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.891594    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:14.891632 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.891623    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:14.891632 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.891633    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx66n\" (UniqueName: \"kubernetes.io/projected/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-kube-api-access-lx66n\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:14.891632 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:14.891642    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace3362d-9bb7-46e2-b218-4e7d3e0ed354-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:47:15.369416 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.369373    4428 generic.go:356] "Generic (PLEG): container finished" podID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" containerID="4ad106c9ce07fd90a1994a884773eb13ec19a77c5fa44fb8675dd543be7c005e" exitCode=0
Oct 31 13:47:15.377219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.369436    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"
Oct 31 13:47:15.377219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.369495    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp" event={"ID":"ace3362d-9bb7-46e2-b218-4e7d3e0ed354","Type":"ContainerDied","Data":"4ad106c9ce07fd90a1994a884773eb13ec19a77c5fa44fb8675dd543be7c005e"}
Oct 31 13:47:15.377219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.369590    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp" event={"ID":"ace3362d-9bb7-46e2-b218-4e7d3e0ed354","Type":"ContainerDied","Data":"0e7a39020deceac4aaa5605ca274062fc654a2415fadc829492dc15274870ad1"}
Oct 31 13:47:15.377219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.369621    4428 scope.go:122] "RemoveContainer" containerID="4ad106c9ce07fd90a1994a884773eb13ec19a77c5fa44fb8675dd543be7c005e"
Oct 31 13:47:15.385765 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.385724    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"]
Oct 31 13:47:15.386989 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.386963    4428 cpu_manager.go:414] "containerMap: removing container" podUID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" containerName="istio-proxy"
Oct 31 13:47:15.386989 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.386990    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" containerName="istio-proxy"
Oct 31 13:47:15.387258 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.387243    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" containerName="istio-proxy"
Oct 31 13:47:15.387327 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.387268    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" containerName="istio-proxy"
Oct 31 13:47:15.387327 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.387283    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1af67f4-a0eb-402b-8453-7cb5ab463ab1" containerName="istio-proxy"
Oct 31 13:47:15.387816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.387796    4428 scope.go:122] "RemoveContainer" containerID="2a73b302ef8ff6424a483a0adfe6e5967a7ec5a467db474c3b2bcd12b1ad5517"
Oct 31 13:47:15.393909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.393388    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 13:47:15.412383 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.412333    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-97927b-serving-cert\""
Oct 31 13:47:15.419738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.419701    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"]
Oct 31 13:47:15.437014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.436977    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"]
Oct 31 13:47:15.438812 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.438648    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0d835b-openshift-default-55fc778f97-ljrdp"]
Oct 31 13:47:15.442082 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.442055    4428 scope.go:122] "RemoveContainer" containerID="4ad106c9ce07fd90a1994a884773eb13ec19a77c5fa44fb8675dd543be7c005e"
Oct 31 13:47:15.442559 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:47:15.442537    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad106c9ce07fd90a1994a884773eb13ec19a77c5fa44fb8675dd543be7c005e\": container with ID starting with 4ad106c9ce07fd90a1994a884773eb13ec19a77c5fa44fb8675dd543be7c005e not found: ID does not exist" containerID="4ad106c9ce07fd90a1994a884773eb13ec19a77c5fa44fb8675dd543be7c005e"
Oct 31 13:47:15.442682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.442570    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad106c9ce07fd90a1994a884773eb13ec19a77c5fa44fb8675dd543be7c005e"} err="failed to get container status \"4ad106c9ce07fd90a1994a884773eb13ec19a77c5fa44fb8675dd543be7c005e\": rpc error: code = NotFound desc = could not find container \"4ad106c9ce07fd90a1994a884773eb13ec19a77c5fa44fb8675dd543be7c005e\": container with ID starting with 4ad106c9ce07fd90a1994a884773eb13ec19a77c5fa44fb8675dd543be7c005e not found: ID does not exist"
Oct 31 13:47:15.442682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.442597    4428 scope.go:122] "RemoveContainer" containerID="2a73b302ef8ff6424a483a0adfe6e5967a7ec5a467db474c3b2bcd12b1ad5517"
Oct 31 13:47:15.443026 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:47:15.443004    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a73b302ef8ff6424a483a0adfe6e5967a7ec5a467db474c3b2bcd12b1ad5517\": container with ID starting with 2a73b302ef8ff6424a483a0adfe6e5967a7ec5a467db474c3b2bcd12b1ad5517 not found: ID does not exist" containerID="2a73b302ef8ff6424a483a0adfe6e5967a7ec5a467db474c3b2bcd12b1ad5517"
Oct 31 13:47:15.443105 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.443033    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a73b302ef8ff6424a483a0adfe6e5967a7ec5a467db474c3b2bcd12b1ad5517"} err="failed to get container status \"2a73b302ef8ff6424a483a0adfe6e5967a7ec5a467db474c3b2bcd12b1ad5517\": rpc error: code = NotFound desc = could not find container \"2a73b302ef8ff6424a483a0adfe6e5967a7ec5a467db474c3b2bcd12b1ad5517\": container with ID starting with 2a73b302ef8ff6424a483a0adfe6e5967a7ec5a467db474c3b2bcd12b1ad5517 not found: ID does not exist"
Oct 31 13:47:15.496941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.496891    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:15.515167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.515085    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b22227-029e-4dff-8530-b5afcdbf58dd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q\" (UID: \"a9b22227-029e-4dff-8530-b5afcdbf58dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 13:47:15.515436 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.515193    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9b22227-029e-4dff-8530-b5afcdbf58dd-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q\" (UID: \"a9b22227-029e-4dff-8530-b5afcdbf58dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 13:47:15.515436 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.515413    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlf87\" (UniqueName: \"kubernetes.io/projected/a9b22227-029e-4dff-8530-b5afcdbf58dd-kube-api-access-dlf87\") pod \"maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q\" (UID: \"a9b22227-029e-4dff-8530-b5afcdbf58dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 13:47:15.515690 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.515493    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9b22227-029e-4dff-8530-b5afcdbf58dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q\" (UID: \"a9b22227-029e-4dff-8530-b5afcdbf58dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 13:47:15.619289 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.619223    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlf87\" (UniqueName: \"kubernetes.io/projected/a9b22227-029e-4dff-8530-b5afcdbf58dd-kube-api-access-dlf87\") pod \"maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q\" (UID: \"a9b22227-029e-4dff-8530-b5afcdbf58dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 13:47:15.619564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.619332    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9b22227-029e-4dff-8530-b5afcdbf58dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q\" (UID: \"a9b22227-029e-4dff-8530-b5afcdbf58dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 13:47:15.619564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.619514    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b22227-029e-4dff-8530-b5afcdbf58dd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q\" (UID: \"a9b22227-029e-4dff-8530-b5afcdbf58dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 13:47:15.619727 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.619562    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9b22227-029e-4dff-8530-b5afcdbf58dd-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q\" (UID: \"a9b22227-029e-4dff-8530-b5afcdbf58dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 13:47:15.639915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.639852    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b22227-029e-4dff-8530-b5afcdbf58dd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q\" (UID: \"a9b22227-029e-4dff-8530-b5afcdbf58dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 13:47:15.659788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.659751    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9b22227-029e-4dff-8530-b5afcdbf58dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q\" (UID: \"a9b22227-029e-4dff-8530-b5afcdbf58dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 13:47:15.661967 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.661933    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9b22227-029e-4dff-8530-b5afcdbf58dd-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q\" (UID: \"a9b22227-029e-4dff-8530-b5afcdbf58dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 13:47:15.662178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.662158    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlf87\" (UniqueName: \"kubernetes.io/projected/a9b22227-029e-4dff-8530-b5afcdbf58dd-kube-api-access-dlf87\") pod \"maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q\" (UID: \"a9b22227-029e-4dff-8530-b5afcdbf58dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 13:47:15.667017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.666430    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:15.732443 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.732404    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"]
Oct 31 13:47:15.734154 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.733830    4428 cpu_manager.go:414] "containerMap: removing container" podUID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" containerName="istio-proxy"
Oct 31 13:47:15.734154 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.733859    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" containerName="istio-proxy"
Oct 31 13:47:15.739400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.739375    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"
Oct 31 13:47:15.747479 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.747447    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"]
Oct 31 13:47:15.747640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.747613    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 13:47:15.751060 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.751022    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-974711-dockercfg-lctt8\""
Oct 31 13:47:15.815755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.815702    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-974711\""
Oct 31 13:47:15.834041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.832326    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2zr2\" (UniqueName: \"kubernetes.io/projected/6bfa6cd9-13c7-4201-9a1f-96877ae4288c-kube-api-access-l2zr2\") pod \"payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs\" (UID: \"6bfa6cd9-13c7-4201-9a1f-96877ae4288c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"
Oct 31 13:47:15.834041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.832766    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfa6cd9-13c7-4201-9a1f-96877ae4288c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs\" (UID: \"6bfa6cd9-13c7-4201-9a1f-96877ae4288c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"
Oct 31 13:47:15.862331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:15.858760    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:16.002755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:16.002543    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfa6cd9-13c7-4201-9a1f-96877ae4288c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs\" (UID: \"6bfa6cd9-13c7-4201-9a1f-96877ae4288c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"
Oct 31 13:47:16.002755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:16.002707    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2zr2\" (UniqueName: \"kubernetes.io/projected/6bfa6cd9-13c7-4201-9a1f-96877ae4288c-kube-api-access-l2zr2\") pod \"payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs\" (UID: \"6bfa6cd9-13c7-4201-9a1f-96877ae4288c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"
Oct 31 13:47:16.008591 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:16.003836    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfa6cd9-13c7-4201-9a1f-96877ae4288c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs\" (UID: \"6bfa6cd9-13c7-4201-9a1f-96877ae4288c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"
Oct 31 13:47:16.095922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:16.090196    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2zr2\" (UniqueName: \"kubernetes.io/projected/6bfa6cd9-13c7-4201-9a1f-96877ae4288c-kube-api-access-l2zr2\") pod \"payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs\" (UID: \"6bfa6cd9-13c7-4201-9a1f-96877ae4288c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"
Oct 31 13:47:16.107575 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:47:16.106294    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b22227_029e_4dff_8530_b5afcdbf58dd.slice/crio-bef487076e2143d9a65f2e27d47f8095f99ea485d9804d09088f0cf202deddb8 WatchSource:0}: Error finding container bef487076e2143d9a65f2e27d47f8095f99ea485d9804d09088f0cf202deddb8: Status 404 returned error can't find the container with id bef487076e2143d9a65f2e27d47f8095f99ea485d9804d09088f0cf202deddb8
Oct 31 13:47:16.108223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:16.108188    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"]
Oct 31 13:47:16.369396 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:16.369300    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"
Oct 31 13:47:16.383707 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:16.383658    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q" event={"ID":"a9b22227-029e-4dff-8530-b5afcdbf58dd","Type":"ContainerStarted","Data":"bef487076e2143d9a65f2e27d47f8095f99ea485d9804d09088f0cf202deddb8"}
Oct 31 13:47:16.759686 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:16.758967    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace3362d-9bb7-46e2-b218-4e7d3e0ed354" path="/var/lib/kubelet/pods/ace3362d-9bb7-46e2-b218-4e7d3e0ed354/volumes"
Oct 31 13:47:16.765811 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:16.765784    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"]
Oct 31 13:47:16.766194 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:47:16.766172    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfa6cd9_13c7_4201_9a1f_96877ae4288c.slice/crio-9a45d56fd6194b74b9766963e1fa8fdc781ac9acd4726913b6a3f55cc7537620 WatchSource:0}: Error finding container 9a45d56fd6194b74b9766963e1fa8fdc781ac9acd4726913b6a3f55cc7537620: Status 404 returned error can't find the container with id 9a45d56fd6194b74b9766963e1fa8fdc781ac9acd4726913b6a3f55cc7537620
Oct 31 13:47:17.283152 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:17.283118    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:17.393504 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:17.393457    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs" event={"ID":"6bfa6cd9-13c7-4201-9a1f-96877ae4288c","Type":"ContainerStarted","Data":"5e133d39a91861a42f5cbeeff03d38c160409a1e61d226e4217d937ee98ae362"}
Oct 31 13:47:17.394074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:17.393528    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs" event={"ID":"6bfa6cd9-13c7-4201-9a1f-96877ae4288c","Type":"ContainerStarted","Data":"9a45d56fd6194b74b9766963e1fa8fdc781ac9acd4726913b6a3f55cc7537620"}
Oct 31 13:47:17.394074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:17.393570    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"
Oct 31 13:47:17.395493 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:17.395466    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q" event={"ID":"a9b22227-029e-4dff-8530-b5afcdbf58dd","Type":"ContainerStarted","Data":"6e6912ed97f019cb3c0a1c78641d8af0f1df63b5b875d1ec56fee03a18ac9f43"}
Oct 31 13:47:17.395644 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:17.395523    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 13:47:17.418082 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:17.418034    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs" podStartSLOduration=2.418019848 podStartE2EDuration="2.418019848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:17.415219767 +0000 UTC m=+2634.813568387" watchObservedRunningTime="2026-10-31 13:47:17.418019848 +0000 UTC m=+2634.816368471"
Oct 31 13:47:17.444119 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:17.444063    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q" podStartSLOduration=1.929037205 podStartE2EDuration="2.444046203s" totalImagesPullingTime="515.008998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:17.44036195 +0000 UTC m=+2634.838710574" watchObservedRunningTime="2026-10-31 13:47:17.444046203 +0000 UTC m=+2634.842394805"
Oct 31 13:47:18.190892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:18.190077    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:18.742302 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:47:18.742265    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:47:19.034268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:19.034018    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:19.909894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:19.901230    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:19.909894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:19.909563    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:19.950539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:19.950507    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:20.432137 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:20.432102    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:47:20.876705 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:20.870476    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:21.730294 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:21.727266    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:22.716480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:22.716442    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:23.408421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:23.408391    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 13:47:23.516485 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:23.516446    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:24.412977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:24.411167    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:25.237623 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:25.237584    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:25.372982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:25.372951    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"
Oct 31 13:47:26.109588 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:26.105512    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:26.830510 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:26.830428    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:27.651661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:27.651629    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:28.421042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:28.418607    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:28.422636 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:28.422138    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"
Oct 31 13:47:29.206897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:29.202552    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:29.952076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:29.952039    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:30.806827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:30.806784    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:30.974893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:30.974645    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:31.872892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:31.872438    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:32.704130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:32.704094    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:33.602502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:33.602461    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:34.396431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.396402    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"]
Oct 31 13:47:34.404666 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.404639    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 13:47:34.424022 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.419664    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-974711-serving-cert\""
Oct 31 13:47:34.427439 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.427409    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"]
Oct 31 13:47:34.507086 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.507051    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75240edd-b28b-4986-992b-6557ad0f41be-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps\" (UID: \"75240edd-b28b-4986-992b-6557ad0f41be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 13:47:34.507339 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.507308    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75240edd-b28b-4986-992b-6557ad0f41be-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps\" (UID: \"75240edd-b28b-4986-992b-6557ad0f41be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 13:47:34.507433 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.507375    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnlz8\" (UniqueName: \"kubernetes.io/projected/75240edd-b28b-4986-992b-6557ad0f41be-kube-api-access-dnlz8\") pod \"maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps\" (UID: \"75240edd-b28b-4986-992b-6557ad0f41be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 13:47:34.507433 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.507418    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75240edd-b28b-4986-992b-6557ad0f41be-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps\" (UID: \"75240edd-b28b-4986-992b-6557ad0f41be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 13:47:34.523440 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.523408    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:34.612960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.611220    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75240edd-b28b-4986-992b-6557ad0f41be-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps\" (UID: \"75240edd-b28b-4986-992b-6557ad0f41be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 13:47:34.612960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.611294    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnlz8\" (UniqueName: \"kubernetes.io/projected/75240edd-b28b-4986-992b-6557ad0f41be-kube-api-access-dnlz8\") pod \"maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps\" (UID: \"75240edd-b28b-4986-992b-6557ad0f41be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 13:47:34.612960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.611323    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75240edd-b28b-4986-992b-6557ad0f41be-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps\" (UID: \"75240edd-b28b-4986-992b-6557ad0f41be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 13:47:34.612960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.611455    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75240edd-b28b-4986-992b-6557ad0f41be-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps\" (UID: \"75240edd-b28b-4986-992b-6557ad0f41be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 13:47:34.620557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.620523    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75240edd-b28b-4986-992b-6557ad0f41be-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps\" (UID: \"75240edd-b28b-4986-992b-6557ad0f41be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 13:47:34.687772 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.669179    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75240edd-b28b-4986-992b-6557ad0f41be-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps\" (UID: \"75240edd-b28b-4986-992b-6557ad0f41be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 13:47:34.687772 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.679740    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnlz8\" (UniqueName: \"kubernetes.io/projected/75240edd-b28b-4986-992b-6557ad0f41be-kube-api-access-dnlz8\") pod \"maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps\" (UID: \"75240edd-b28b-4986-992b-6557ad0f41be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 13:47:34.691202 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.690802    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75240edd-b28b-4986-992b-6557ad0f41be-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps\" (UID: \"75240edd-b28b-4986-992b-6557ad0f41be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 13:47:34.709151 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.709122    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:34.735011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:34.734974    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 13:47:35.056900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:35.050992    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"]
Oct 31 13:47:35.626128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:35.626092    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:35.670391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:35.670351    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps" event={"ID":"75240edd-b28b-4986-992b-6557ad0f41be","Type":"ContainerStarted","Data":"f4025b6a5f0669236d4d6663f7e7fc6022a2769ab02cd0c4a985a3835cc71514"}
Oct 31 13:47:35.795914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:35.792491    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:36.529901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:36.527987    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:36.680227 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:36.680189    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps" event={"ID":"75240edd-b28b-4986-992b-6557ad0f41be","Type":"ContainerStarted","Data":"51b1ab3e2c4a5ffdfa9097a7a00af21bee7d3d146684595f0d07b934b6d9d985"}
Oct 31 13:47:36.680673 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:36.680282    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 13:47:36.706047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:36.705993    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps" podStartSLOduration=2.178044463 podStartE2EDuration="2.70597752s" totalImagesPullingTime="527.933057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:36.701027512 +0000 UTC m=+2654.099376136" watchObservedRunningTime="2026-10-31 13:47:36.70597752 +0000 UTC m=+2654.104326143"
Oct 31 13:47:37.430218 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:37.430180    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:38.021483 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:38.020504    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:38.311953 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:38.308940    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:38.532481 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:38.532410    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:38.895259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:38.895214    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"]
Oct 31 13:47:38.900336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:38.900312    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:38.925265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:38.925225    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"]
Oct 31 13:47:38.949172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:38.949129    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fghl\" (UniqueName: \"kubernetes.io/projected/4608de93-17e1-4803-a234-bfb4e24e559f-kube-api-access-7fghl\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:38.949384 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:38.949215    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-istio-envoy\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:38.949384 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:38.949327    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-workload-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:38.949520 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:38.949501    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4608de93-17e1-4803-a234-bfb4e24e559f-istio-token\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:38.949599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:38.949582    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-istio-data\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:38.949646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:38.949627    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-credential-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:38.949688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:38.949669    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4608de93-17e1-4803-a234-bfb4e24e559f-istiod-ca-cert\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:38.949813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:38.949789    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4608de93-17e1-4803-a234-bfb4e24e559f-istio-podinfo\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:38.950262 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:38.950237    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-workload-certs\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:38.963709 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:38.963672    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-599291-openshift-default-dockercfg-9db8q\""
Oct 31 13:47:38.974749 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:38.974723    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:39.093893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.083409    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4608de93-17e1-4803-a234-bfb4e24e559f-istio-token\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.093893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.083505    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-istio-data\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.093893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.083595    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-credential-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.093893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.083636    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4608de93-17e1-4803-a234-bfb4e24e559f-istiod-ca-cert\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.093893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.083670    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4608de93-17e1-4803-a234-bfb4e24e559f-istio-podinfo\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.093893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.083731    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-workload-certs\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.093893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.083830    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fghl\" (UniqueName: \"kubernetes.io/projected/4608de93-17e1-4803-a234-bfb4e24e559f-kube-api-access-7fghl\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.093893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.083902    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-istio-envoy\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.093893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.084007    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-workload-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.093893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.084556    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-workload-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.104947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.095397    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-istio-data\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.104947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.095689    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-credential-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.104947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.096399    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4608de93-17e1-4803-a234-bfb4e24e559f-istiod-ca-cert\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.104947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.100031    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4608de93-17e1-4803-a234-bfb4e24e559f-istio-podinfo\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.104947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.100309    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-workload-certs\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.135148 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.124474    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-istio-envoy\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.135148 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.128450    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4608de93-17e1-4803-a234-bfb4e24e559f-istio-token\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.142426 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.142261    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:39.166949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.163274    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fghl\" (UniqueName: \"kubernetes.io/projected/4608de93-17e1-4803-a234-bfb4e24e559f-kube-api-access-7fghl\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.225713 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.225432    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:39.476379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.472436    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:39.559721 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.559692    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"]
Oct 31 13:47:39.559923 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:47:39.559789    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4608de93_17e1_4803_a234_bfb4e24e559f.slice/crio-ff698a69b53a75d8b7707e36a502318689cfcf387f6f185da15890fe709fe36e WatchSource:0}: Error finding container ff698a69b53a75d8b7707e36a502318689cfcf387f6f185da15890fe709fe36e: Status 404 returned error can't find the container with id ff698a69b53a75d8b7707e36a502318689cfcf387f6f185da15890fe709fe36e
Oct 31 13:47:39.562908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.562849    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:47:39.563005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.562958    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:47:39.563005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.562997    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:47:39.710458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.710424    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9" event={"ID":"4608de93-17e1-4803-a234-bfb4e24e559f","Type":"ContainerStarted","Data":"20dc94b7e65fdb678e77e620cd0146015123b97adce99691977fa9c6063f28a4"}
Oct 31 13:47:39.710693 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.710493    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9" event={"ID":"4608de93-17e1-4803-a234-bfb4e24e559f","Type":"ContainerStarted","Data":"ff698a69b53a75d8b7707e36a502318689cfcf387f6f185da15890fe709fe36e"}
Oct 31 13:47:39.736186 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:39.736069    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9" podStartSLOduration=1.736049449 podStartE2EDuration="1.736049449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:39.734492131 +0000 UTC m=+2657.132840809" watchObservedRunningTime="2026-10-31 13:47:39.736049449 +0000 UTC m=+2657.134398074"
Oct 31 13:47:40.230776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:40.230739    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:40.237161 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:40.237136    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:40.430893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:40.426826    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:40.719925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:40.719886    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:40.721038 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:40.721017    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:47:41.674113 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:41.674079    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:42.138617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:42.138585    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:42.325417 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:42.325375    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:42.485550 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:42.485503    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:42.656841 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:42.656806    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:42.694874 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:42.694848    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 13:47:43.334422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:43.334386    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"]
Oct 31 13:47:43.340258 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:43.340230    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"
Oct 31 13:47:43.345304 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:43.345283    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-599291\""
Oct 31 13:47:43.357252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:43.357223    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:43.367242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:43.365763    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"]
Oct 31 13:47:43.409320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:43.409275    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7t52\" (UniqueName: \"kubernetes.io/projected/71b06a52-1d53-48e7-a5bd-b25f92a89e91-kube-api-access-x7t52\") pod \"payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv\" (UID: \"71b06a52-1d53-48e7-a5bd-b25f92a89e91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"
Oct 31 13:47:43.409516 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:43.409459    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b06a52-1d53-48e7-a5bd-b25f92a89e91-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv\" (UID: \"71b06a52-1d53-48e7-a5bd-b25f92a89e91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"
Oct 31 13:47:43.511211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:43.511176    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7t52\" (UniqueName: \"kubernetes.io/projected/71b06a52-1d53-48e7-a5bd-b25f92a89e91-kube-api-access-x7t52\") pod \"payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv\" (UID: \"71b06a52-1d53-48e7-a5bd-b25f92a89e91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"
Oct 31 13:47:43.511418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:43.511353    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b06a52-1d53-48e7-a5bd-b25f92a89e91-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv\" (UID: \"71b06a52-1d53-48e7-a5bd-b25f92a89e91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"
Oct 31 13:47:43.512113 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:43.512089    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b06a52-1d53-48e7-a5bd-b25f92a89e91-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv\" (UID: \"71b06a52-1d53-48e7-a5bd-b25f92a89e91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"
Oct 31 13:47:43.524467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:43.524444    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7t52\" (UniqueName: \"kubernetes.io/projected/71b06a52-1d53-48e7-a5bd-b25f92a89e91-kube-api-access-x7t52\") pod \"payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv\" (UID: \"71b06a52-1d53-48e7-a5bd-b25f92a89e91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"
Oct 31 13:47:43.601951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:43.588628    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:43.661355 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:43.659830    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"
Oct 31 13:47:43.814646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:43.806033    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:44.097650 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:47:44.097618    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b06a52_1d53_48e7_a5bd_b25f92a89e91.slice/crio-99a6f8197c4f2b0e8483c0f509516d967181c01d55806d068c8440f002cbe2e8 WatchSource:0}: Error finding container 99a6f8197c4f2b0e8483c0f509516d967181c01d55806d068c8440f002cbe2e8: Status 404 returned error can't find the container with id 99a6f8197c4f2b0e8483c0f509516d967181c01d55806d068c8440f002cbe2e8
Oct 31 13:47:44.098724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:44.098689    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"]
Oct 31 13:47:44.416336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:44.414542    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:44.835973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:44.835937    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv" event={"ID":"71b06a52-1d53-48e7-a5bd-b25f92a89e91","Type":"ContainerStarted","Data":"8b699afc29f2d8d9241147d585c8407538f20577735eb3be8321b037c9fc1c3c"}
Oct 31 13:47:44.836427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:44.835998    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv" event={"ID":"71b06a52-1d53-48e7-a5bd-b25f92a89e91","Type":"ContainerStarted","Data":"99a6f8197c4f2b0e8483c0f509516d967181c01d55806d068c8440f002cbe2e8"}
Oct 31 13:47:44.836427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:44.836066    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"
Oct 31 13:47:44.859974 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:44.859910    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv" podStartSLOduration=1.859895305 podStartE2EDuration="1.859895305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:44.855410799 +0000 UTC m=+2662.253759426" watchObservedRunningTime="2026-10-31 13:47:44.859895305 +0000 UTC m=+2662.258243933"
Oct 31 13:47:45.211361 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:45.207832    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:46.131815 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:46.131744    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:47.120900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:47.118542    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:47.919148 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:47.916609    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:48.742678 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:47:48.742636    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:47:48.801681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:48.800972    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:49.761894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:49.756465    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:50.518358 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:50.518310    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:51.349959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:51.347257    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:52.109903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:52.108080    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:53.042378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:53.042341    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:54.016494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:54.016454    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:54.824264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:54.824229    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:55.903086 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:55.903055    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"
Oct 31 13:47:56.218951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:56.216101    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:56.556382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:56.556283    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:56.820224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:56.820127    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:57.690893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:57.688187    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:57.898447 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:57.898386    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:58.638555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:58.638501    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:47:59.574479 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:47:59.574443    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:00.068908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:00.068753    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:00.505925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:00.505894    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:00.666130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:00.666098    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:01.080903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:01.077746    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:01.318895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:01.318228    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:01.796852 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:01.796821    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:02.085044 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.084963    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:02.131162 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.131112    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"]
Oct 31 13:48:02.136120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.136098    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"
Oct 31 13:48:02.143746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.143718    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-35d13e\""
Oct 31 13:48:02.143970 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.143848    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-35d13e-dockercfg-lj29q\""
Oct 31 13:48:02.177919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.147637    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"]
Oct 31 13:48:02.193658 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.187038    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"]
Oct 31 13:48:02.193658 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.192123    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"
Oct 31 13:48:02.201856 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.201823    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"]
Oct 31 13:48:02.360955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.320931    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjnnw\" (UniqueName: \"kubernetes.io/projected/bc620c07-7229-4b69-9686-6ae7786b869f-kube-api-access-sjnnw\") pod \"payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs\" (UID: \"bc620c07-7229-4b69-9686-6ae7786b869f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"
Oct 31 13:48:02.361331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.361299    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jzdp\" (UniqueName: \"kubernetes.io/projected/81d7c2b0-44d4-4b9f-8000-aaf84a56c722-kube-api-access-7jzdp\") pod \"payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh\" (UID: \"81d7c2b0-44d4-4b9f-8000-aaf84a56c722\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"
Oct 31 13:48:02.361535 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.361518    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d7c2b0-44d4-4b9f-8000-aaf84a56c722-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh\" (UID: \"81d7c2b0-44d4-4b9f-8000-aaf84a56c722\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"
Oct 31 13:48:02.365890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.361784    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc620c07-7229-4b69-9686-6ae7786b869f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs\" (UID: \"bc620c07-7229-4b69-9686-6ae7786b869f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"
Oct 31 13:48:02.385105 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.381132    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:02.514396 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.488187    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjnnw\" (UniqueName: \"kubernetes.io/projected/bc620c07-7229-4b69-9686-6ae7786b869f-kube-api-access-sjnnw\") pod \"payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs\" (UID: \"bc620c07-7229-4b69-9686-6ae7786b869f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"
Oct 31 13:48:02.515182 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.515148    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jzdp\" (UniqueName: \"kubernetes.io/projected/81d7c2b0-44d4-4b9f-8000-aaf84a56c722-kube-api-access-7jzdp\") pod \"payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh\" (UID: \"81d7c2b0-44d4-4b9f-8000-aaf84a56c722\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"
Oct 31 13:48:02.515468 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.515446    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d7c2b0-44d4-4b9f-8000-aaf84a56c722-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh\" (UID: \"81d7c2b0-44d4-4b9f-8000-aaf84a56c722\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"
Oct 31 13:48:02.517226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.516963    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d7c2b0-44d4-4b9f-8000-aaf84a56c722-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh\" (UID: \"81d7c2b0-44d4-4b9f-8000-aaf84a56c722\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"
Oct 31 13:48:02.517566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.517544    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc620c07-7229-4b69-9686-6ae7786b869f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs\" (UID: \"bc620c07-7229-4b69-9686-6ae7786b869f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"
Oct 31 13:48:02.523272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.518489    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc620c07-7229-4b69-9686-6ae7786b869f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs\" (UID: \"bc620c07-7229-4b69-9686-6ae7786b869f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"
Oct 31 13:48:02.579731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.579700    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jzdp\" (UniqueName: \"kubernetes.io/projected/81d7c2b0-44d4-4b9f-8000-aaf84a56c722-kube-api-access-7jzdp\") pod \"payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh\" (UID: \"81d7c2b0-44d4-4b9f-8000-aaf84a56c722\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"
Oct 31 13:48:02.590700 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.590641    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjnnw\" (UniqueName: \"kubernetes.io/projected/bc620c07-7229-4b69-9686-6ae7786b869f-kube-api-access-sjnnw\") pod \"payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs\" (UID: \"bc620c07-7229-4b69-9686-6ae7786b869f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"
Oct 31 13:48:02.787411 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.787297    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"
Oct 31 13:48:02.812098 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.812042    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"
Oct 31 13:48:03.009982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:02.996102    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:03.111614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:03.111478    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"]
Oct 31 13:48:03.115896 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:48:03.115845    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc620c07_7229_4b69_9686_6ae7786b869f.slice/crio-e3a754c80634416bd1584d13a954f21b5b4a69d24058f3c3f02f0d08b4533033 WatchSource:0}: Error finding container e3a754c80634416bd1584d13a954f21b5b4a69d24058f3c3f02f0d08b4533033: Status 404 returned error can't find the container with id e3a754c80634416bd1584d13a954f21b5b4a69d24058f3c3f02f0d08b4533033
Oct 31 13:48:03.239521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:03.239490    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"]
Oct 31 13:48:03.248019 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:48:03.247983    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d7c2b0_44d4_4b9f_8000_aaf84a56c722.slice/crio-92edce4e11f4266ab40faa3100e1470d340b8b70c216331adf7d85560cb91b59 WatchSource:0}: Error finding container 92edce4e11f4266ab40faa3100e1470d340b8b70c216331adf7d85560cb91b59: Status 404 returned error can't find the container with id 92edce4e11f4266ab40faa3100e1470d340b8b70c216331adf7d85560cb91b59
Oct 31 13:48:03.449850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:03.449810    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs" event={"ID":"bc620c07-7229-4b69-9686-6ae7786b869f","Type":"ContainerStarted","Data":"c79578ee5cdd7307fb214c52486ca078f36f4b2e6fad0f846d74021b2edc5378"}
Oct 31 13:48:03.450062 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:03.449884    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs" event={"ID":"bc620c07-7229-4b69-9686-6ae7786b869f","Type":"ContainerStarted","Data":"e3a754c80634416bd1584d13a954f21b5b4a69d24058f3c3f02f0d08b4533033"}
Oct 31 13:48:03.450062 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:03.449914    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"
Oct 31 13:48:03.451919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:03.451886    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh" event={"ID":"81d7c2b0-44d4-4b9f-8000-aaf84a56c722","Type":"ContainerStarted","Data":"b28b7442c1fc415aecd36ad7a5644853443e4be61d21f874b31fd618ab5997c4"}
Oct 31 13:48:03.452081 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:03.451942    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh" event={"ID":"81d7c2b0-44d4-4b9f-8000-aaf84a56c722","Type":"ContainerStarted","Data":"92edce4e11f4266ab40faa3100e1470d340b8b70c216331adf7d85560cb91b59"}
Oct 31 13:48:03.452081 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:03.451981    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"
Oct 31 13:48:03.477738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:03.477679    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs" podStartSLOduration=1.477663232 podStartE2EDuration="1.477663232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:03.476990237 +0000 UTC m=+2680.875338863" watchObservedRunningTime="2026-10-31 13:48:03.477663232 +0000 UTC m=+2680.876011887"
Oct 31 13:48:03.507047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:03.506980    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh" podStartSLOduration=1.5069606260000001 podStartE2EDuration="1.506960626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:03.500827174 +0000 UTC m=+2680.899175799" watchObservedRunningTime="2026-10-31 13:48:03.506960626 +0000 UTC m=+2680.905309251"
Oct 31 13:48:03.948715 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:03.948652    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:04.111039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:04.111004    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:04.270202 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:04.270168    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:05.166853 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:05.166815    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:05.351901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:05.348808    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:06.062851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.062753    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:06.412613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.412516    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:06.477786 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.477751    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"]
Oct 31 13:48:06.482987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.482956    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"
Oct 31 13:48:06.488149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.487974    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"]
Oct 31 13:48:06.492182 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.492062    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-14ba5a\""
Oct 31 13:48:06.492373 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.492200    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-14ba5a-dockercfg-wqx62\""
Oct 31 13:48:06.506635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.506604    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"
Oct 31 13:48:06.508230 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.508131    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"]
Oct 31 13:48:06.573258 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.572213    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"]
Oct 31 13:48:06.687976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.662171    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:06.717911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.704449    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjklx\" (UniqueName: \"kubernetes.io/projected/bab754b1-4f25-4274-a326-579b5326b39a-kube-api-access-vjklx\") pod \"payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f\" (UID: \"bab754b1-4f25-4274-a326-579b5326b39a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"
Oct 31 13:48:06.717911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.704649    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9zks\" (UniqueName: \"kubernetes.io/projected/461e6055-9080-4aab-8f66-43139b5b9cc4-kube-api-access-c9zks\") pod \"payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8\" (UID: \"461e6055-9080-4aab-8f66-43139b5b9cc4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"
Oct 31 13:48:06.717911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.704758    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461e6055-9080-4aab-8f66-43139b5b9cc4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8\" (UID: \"461e6055-9080-4aab-8f66-43139b5b9cc4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"
Oct 31 13:48:06.717911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.704843    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bab754b1-4f25-4274-a326-579b5326b39a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f\" (UID: \"bab754b1-4f25-4274-a326-579b5326b39a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"
Oct 31 13:48:06.816753 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.816636    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9zks\" (UniqueName: \"kubernetes.io/projected/461e6055-9080-4aab-8f66-43139b5b9cc4-kube-api-access-c9zks\") pod \"payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8\" (UID: \"461e6055-9080-4aab-8f66-43139b5b9cc4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"
Oct 31 13:48:06.816989 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.816760    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461e6055-9080-4aab-8f66-43139b5b9cc4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8\" (UID: \"461e6055-9080-4aab-8f66-43139b5b9cc4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"
Oct 31 13:48:06.816989 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.816831    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bab754b1-4f25-4274-a326-579b5326b39a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f\" (UID: \"bab754b1-4f25-4274-a326-579b5326b39a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"
Oct 31 13:48:06.817120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.817062    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjklx\" (UniqueName: \"kubernetes.io/projected/bab754b1-4f25-4274-a326-579b5326b39a-kube-api-access-vjklx\") pod \"payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f\" (UID: \"bab754b1-4f25-4274-a326-579b5326b39a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"
Oct 31 13:48:06.824013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.823736    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bab754b1-4f25-4274-a326-579b5326b39a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f\" (UID: \"bab754b1-4f25-4274-a326-579b5326b39a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"
Oct 31 13:48:06.835734 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.834999    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461e6055-9080-4aab-8f66-43139b5b9cc4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8\" (UID: \"461e6055-9080-4aab-8f66-43139b5b9cc4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"
Oct 31 13:48:06.840525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.840496    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjklx\" (UniqueName: \"kubernetes.io/projected/bab754b1-4f25-4274-a326-579b5326b39a-kube-api-access-vjklx\") pod \"payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f\" (UID: \"bab754b1-4f25-4274-a326-579b5326b39a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"
Oct 31 13:48:06.841201 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.841179    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"
Oct 31 13:48:06.842732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.842711    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9zks\" (UniqueName: \"kubernetes.io/projected/461e6055-9080-4aab-8f66-43139b5b9cc4-kube-api-access-c9zks\") pod \"payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8\" (UID: \"461e6055-9080-4aab-8f66-43139b5b9cc4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"
Oct 31 13:48:06.883257 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.883201    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"
Oct 31 13:48:06.893447 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.893409    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:48:06.893623 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:06.893471    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5" podUID="4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:48:07.100061 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:07.099994    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:07.177375 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:07.177283    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"]
Oct 31 13:48:07.292527 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:07.292490    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"]
Oct 31 13:48:07.293523 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:48:07.293480    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461e6055_9080_4aab_8f66_43139b5b9cc4.slice/crio-5ee127456fca8d07a22c9856ea8943f29305754c59bc6ca92f95b959c9de4a44 WatchSource:0}: Error finding container 5ee127456fca8d07a22c9856ea8943f29305754c59bc6ca92f95b959c9de4a44: Status 404 returned error can't find the container with id 5ee127456fca8d07a22c9856ea8943f29305754c59bc6ca92f95b959c9de4a44
Oct 31 13:48:07.784166 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:07.784131    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f" event={"ID":"bab754b1-4f25-4274-a326-579b5326b39a","Type":"ContainerStarted","Data":"30d972a1ed8bc132c0ed4a9aa8ce5e45564c4be675592c69bc7fff839b243452"}
Oct 31 13:48:07.785947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:07.784193    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f" event={"ID":"bab754b1-4f25-4274-a326-579b5326b39a","Type":"ContainerStarted","Data":"dcfd111256fdd5fc98b752da5ac969914fe924e7abb1f6675a649a67bd52bc59"}
Oct 31 13:48:07.785947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:07.784275    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"
Oct 31 13:48:07.786915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:07.786884    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8" event={"ID":"461e6055-9080-4aab-8f66-43139b5b9cc4","Type":"ContainerStarted","Data":"7b681ae63c8fd118ac450541b1dfce433edb2f63b8f9a076af9dcf0bab675d5a"}
Oct 31 13:48:07.787022 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:07.786942    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8" event={"ID":"461e6055-9080-4aab-8f66-43139b5b9cc4","Type":"ContainerStarted","Data":"5ee127456fca8d07a22c9856ea8943f29305754c59bc6ca92f95b959c9de4a44"}
Oct 31 13:48:07.787022 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:07.786973    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"
Oct 31 13:48:07.808287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:07.808231    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f" podStartSLOduration=1.8082158879999999 podStartE2EDuration="1.808215888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:07.806702873 +0000 UTC m=+2685.205051502" watchObservedRunningTime="2026-10-31 13:48:07.808215888 +0000 UTC m=+2685.206564511"
Oct 31 13:48:07.830545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:07.830482    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8" podStartSLOduration=1.830464311 podStartE2EDuration="1.830464311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:07.829579466 +0000 UTC m=+2685.227928092" watchObservedRunningTime="2026-10-31 13:48:07.830464311 +0000 UTC m=+2685.228812935"
Oct 31 13:48:07.921831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:07.921790    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:08.797010 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:08.796915    4428 generic.go:356] "Generic (PLEG): container finished" podID="4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" containerID="694883a8256ff938e1e8e8358a22a3d5b7b1ed9c6480a4188447472456c64804" exitCode=0
Oct 31 13:48:08.798954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:08.798137    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5" event={"ID":"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e","Type":"ContainerDied","Data":"694883a8256ff938e1e8e8358a22a3d5b7b1ed9c6480a4188447472456c64804"}
Oct 31 13:48:08.837890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:08.837482    4428 scope.go:122] "RemoveContainer" containerID="694883a8256ff938e1e8e8358a22a3d5b7b1ed9c6480a4188447472456c64804"
Oct 31 13:48:08.861125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:08.860157    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:48:08.861125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:08.860289    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:48:08.861125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:08.860342    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:48:08.900075 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:08.900027    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:09.804890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:09.792616    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:09.842890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:09.842832    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5" event={"ID":"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e","Type":"ContainerStarted","Data":"5fced279f9bcf70360c81f79e77f24bab3e322b6e96d6969d0c73bee00799e69"}
Oct 31 13:48:10.491175 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:10.491143    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:48:10.492101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:10.492070    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 31 13:48:10.492233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:10.492134    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5" podUID="4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 31 13:48:10.660892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:10.656917    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:11.043628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.043591    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"]
Oct 31 13:48:11.078673 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.077845    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"]
Oct 31 13:48:11.078673 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.077999    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.102049 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.101999    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-credential-socket\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.102233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.102052    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/028e0271-4e2d-48fd-851f-d5e82e774031-istio-token\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.102233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.102117    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-istio-envoy\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.102233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.102147    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/028e0271-4e2d-48fd-851f-d5e82e774031-istio-podinfo\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.102233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.102176    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cw5q\" (UniqueName: \"kubernetes.io/projected/028e0271-4e2d-48fd-851f-d5e82e774031-kube-api-access-5cw5q\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.102233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.102200    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/028e0271-4e2d-48fd-851f-d5e82e774031-istiod-ca-cert\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.102427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.102260    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-workload-certs\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.102427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.102296    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-istio-data\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.102427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.102370    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-workload-socket\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.167896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.156205    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-35b584-openshift-default-dockercfg-ms4g9\""
Oct 31 13:48:11.203761 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.203721    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-istio-data\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.203966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.203886    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-workload-socket\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.204086 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.204063    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-credential-socket\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.204179 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.204106    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/028e0271-4e2d-48fd-851f-d5e82e774031-istio-token\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.204323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.204238    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-istio-data\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.204323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.204260    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-istio-envoy\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.204323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.204306    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-workload-socket\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.204520 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.204313    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/028e0271-4e2d-48fd-851f-d5e82e774031-istio-podinfo\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.204520 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.204401    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cw5q\" (UniqueName: \"kubernetes.io/projected/028e0271-4e2d-48fd-851f-d5e82e774031-kube-api-access-5cw5q\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.204520 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.204415    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-credential-socket\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.204520 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.204449    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/028e0271-4e2d-48fd-851f-d5e82e774031-istiod-ca-cert\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.204728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.204557    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-workload-certs\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.204918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.204896    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-workload-certs\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.205149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.205131    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/028e0271-4e2d-48fd-851f-d5e82e774031-istiod-ca-cert\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.206848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.206826    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-istio-envoy\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.206947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.206932    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/028e0271-4e2d-48fd-851f-d5e82e774031-istio-podinfo\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.218189 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.218153    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/028e0271-4e2d-48fd-851f-d5e82e774031-istio-token\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.220889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.220851    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cw5q\" (UniqueName: \"kubernetes.io/projected/028e0271-4e2d-48fd-851f-d5e82e774031-kube-api-access-5cw5q\") pod \"e2e-inf-b-35b584-openshift-default-556b575787-8lxl6\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") " pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.405952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.405056    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:11.513997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.511090    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 31 13:48:11.513997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.511169    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5" podUID="4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 31 13:48:11.535761 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.534045    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:11.875206 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.875171    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"]
Oct 31 13:48:11.877778 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:48:11.877748    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028e0271_4e2d_48fd_851f_d5e82e774031.slice/crio-7211025c75842926a4f6a5df424cec68d2207244ff7e3b09419baac11a23871c WatchSource:0}: Error finding container 7211025c75842926a4f6a5df424cec68d2207244ff7e3b09419baac11a23871c: Status 404 returned error can't find the container with id 7211025c75842926a4f6a5df424cec68d2207244ff7e3b09419baac11a23871c
Oct 31 13:48:11.881661 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.881612    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:48:11.881812 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.881725    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:48:11.881812 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:11.881775    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:48:12.339949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:12.334059    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:12.491753 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:12.491717    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 31 13:48:12.491954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:12.491772    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5" podUID="4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 31 13:48:12.898272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:12.898224    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6" event={"ID":"028e0271-4e2d-48fd-851f-d5e82e774031","Type":"ContainerStarted","Data":"1831ad23966254404676845fa07450c9301732506db557e6c5e109b5598a5c3c"}
Oct 31 13:48:12.898504 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:12.898284    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6" event={"ID":"028e0271-4e2d-48fd-851f-d5e82e774031","Type":"ContainerStarted","Data":"7211025c75842926a4f6a5df424cec68d2207244ff7e3b09419baac11a23871c"}
Oct 31 13:48:12.930580 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:12.930519    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6" podStartSLOduration=1.9305034779999999 podStartE2EDuration="1.930503478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:12.925804514 +0000 UTC m=+2690.324153137" watchObservedRunningTime="2026-10-31 13:48:12.930503478 +0000 UTC m=+2690.328852099"
Oct 31 13:48:13.404894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:13.404421    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:13.433894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:13.424178    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:13.433894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:13.425458    4428 patch_prober.go:29] interesting pod/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 31 13:48:13.433894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:13.425514    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6" podUID="028e0271-4e2d-48fd-851f-d5e82e774031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 31 13:48:13.498896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:13.498101    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 31 13:48:13.498896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:13.498171    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5" podUID="4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 31 13:48:14.320019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:14.319968    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:14.411357 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:14.411307    4428 patch_prober.go:29] interesting pod/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 31 13:48:14.412074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:14.412028    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6" podUID="028e0271-4e2d-48fd-851f-d5e82e774031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 31 13:48:14.495896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:14.494947    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 31 13:48:14.495896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:14.495010    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5" podUID="4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 31 13:48:14.499971 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:14.499504    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"
Oct 31 13:48:14.499971 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:14.499602    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"
Oct 31 13:48:14.501647 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:14.501623    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:15.222861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:15.222802    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:15.410559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:15.410432    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:15.410559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:15.410539    4428 patch_prober.go:29] interesting pod/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 31 13:48:15.410817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:15.410605    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6" podUID="028e0271-4e2d-48fd-851f-d5e82e774031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 31 13:48:15.506993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:15.506705    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:48:15.507514 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:15.507487    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:48:15.508802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:15.508778    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 13:48:16.411321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:16.411279    4428 patch_prober.go:29] interesting pod/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 31 13:48:16.416535 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:16.416498    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6" podUID="028e0271-4e2d-48fd-851f-d5e82e774031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 31 13:48:16.417293 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:16.416251    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:17.412901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:17.406174    4428 patch_prober.go:29] interesting pod/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 31 13:48:17.412901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:17.406245    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6" podUID="028e0271-4e2d-48fd-851f-d5e82e774031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 31 13:48:17.458891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:17.452531    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:18.413896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:18.413321    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:18.430893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:18.430184    4428 patch_prober.go:29] interesting pod/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 31 13:48:18.430893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:18.430304    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6" podUID="028e0271-4e2d-48fd-851f-d5e82e774031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 31 13:48:18.766761 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:48:18.766714    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:48:18.801603 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:18.801578    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"
Oct 31 13:48:18.843495 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:18.843462    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"
Oct 31 13:48:19.359304 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:19.359264    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:19.425788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:19.425733    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:19.426603 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:19.426580    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:19.427491 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:19.427468    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:48:20.377564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:20.377512    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:21.350530 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:21.350495    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:22.185895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:22.182466    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:23.113514 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:23.113432    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:23.900662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:23.900629    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:24.077730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:24.077666    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:25.180878 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:25.180787    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:26.119396 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:26.119355    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:27.061896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:27.046913    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:27.982553 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:27.982099    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:28.150800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:28.150756    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:29.156708 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:29.153465    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:30.404889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:30.401656    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:31.264726 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:31.264690    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:32.382115 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:32.382083    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:33.018434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:33.018394    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:33.325837 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:33.325693    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:34.177430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:34.177353    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:35.114892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:35.105508    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:35.953070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:35.952961    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:36.712850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:36.712810    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:36.924143 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:36.924087    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:37.710040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:37.710002    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:38.530980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:38.530890    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:39.407424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:39.407379    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:40.248005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:40.247881    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:41.131895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:41.129860    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:42.097830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:42.097796    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:43.008280 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:43.008246    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:43.918602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:43.918566    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:44.803558 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:44.803510    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:45.607686 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:45.607648    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:46.417650 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:46.417607    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:47.281830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:47.281463    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:48.148893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:48.143495    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:48.743066 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:48:48.743023    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:48:49.134524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:49.134428    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:50.113962 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:50.113914    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:50.963120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:50.963083    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:51.795466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:51.795428    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:51.952263 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:51.952223    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:52.850824 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:52.850785    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:53.847829 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:53.847789    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:54.365401 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:54.365364    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:55.502743 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:55.502700    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:56.027915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:56.026940    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:56.540892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:56.531217    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:57.313566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:57.313483    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:58.326894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:58.326106    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:59.099800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:59.099761    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:48:59.935890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:48:59.929390    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:00.831898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:00.831185    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:01.727901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:01.724401    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:02.565691 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:02.560399    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:03.425891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:03.425494    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:04.252839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:04.252580    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:05.088320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:05.086567    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:05.681224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:05.681192    4428 generic.go:356] "Generic (PLEG): container finished" podID="4608de93-17e1-4803-a234-bfb4e24e559f" containerID="20dc94b7e65fdb678e77e620cd0146015123b97adce99691977fa9c6063f28a4" exitCode=0
Oct 31 13:49:05.683487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:05.681263    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9" event={"ID":"4608de93-17e1-4803-a234-bfb4e24e559f","Type":"ContainerDied","Data":"20dc94b7e65fdb678e77e620cd0146015123b97adce99691977fa9c6063f28a4"}
Oct 31 13:49:05.683487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:05.681688    4428 scope.go:122] "RemoveContainer" containerID="20dc94b7e65fdb678e77e620cd0146015123b97adce99691977fa9c6063f28a4"
Oct 31 13:49:05.683914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:05.683610    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:49:05.683914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:05.683676    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:49:05.683914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:05.683712    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:49:05.760465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:05.760427    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": read tcp 10.134.0.2:60896->10.134.0.225:15021: read: connection reset by peer" start-of-body=
Oct 31 13:49:05.760613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:05.760498    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb" podUID="fdbe827a-0f8e-41f0-a10e-0893d404efbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": read tcp 10.134.0.2:60896->10.134.0.225:15021: read: connection reset by peer"
Oct 31 13:49:05.903423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:05.902104    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:06.723897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:06.713369    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:06.751784 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:06.751350    4428 generic.go:356] "Generic (PLEG): container finished" podID="fdbe827a-0f8e-41f0-a10e-0893d404efbe" containerID="e6ee5852d7a7f2ef0706c0175847e8eb8744833445e552a8018fd670b0eec565" exitCode=0
Oct 31 13:49:06.779311 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:06.779203    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb" event={"ID":"fdbe827a-0f8e-41f0-a10e-0893d404efbe","Type":"ContainerDied","Data":"e6ee5852d7a7f2ef0706c0175847e8eb8744833445e552a8018fd670b0eec565"}
Oct 31 13:49:06.779311 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:06.779282    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9" event={"ID":"4608de93-17e1-4803-a234-bfb4e24e559f","Type":"ContainerStarted","Data":"0e0f127e614ce872d8061097c9eb740176d4be08ebfff71d0c4d52491bc13647"}
Oct 31 13:49:06.780391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:06.780368    4428 scope.go:122] "RemoveContainer" containerID="e6ee5852d7a7f2ef0706c0175847e8eb8744833445e552a8018fd670b0eec565"
Oct 31 13:49:06.782296 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:06.782260    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:49:06.782374 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:06.782363    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:49:06.782430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:06.782405    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:49:07.230587 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:07.230554    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:49:07.231793 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:07.231725    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 31 13:49:07.231988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:07.231800    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9" podUID="4608de93-17e1-4803-a234-bfb4e24e559f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 31 13:49:07.667634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:07.667540    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:07.778809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:07.778770    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb" event={"ID":"fdbe827a-0f8e-41f0-a10e-0893d404efbe","Type":"ContainerStarted","Data":"ad32ac23a1e52aad0193f0a3066df81caf88785204b17ba2243c13ff8f27bda2"}
Oct 31 13:49:07.887685 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:07.887237    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:49:07.887685 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:07.887370    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 31 13:49:07.887685 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:07.887411    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb" podUID="fdbe827a-0f8e-41f0-a10e-0893d404efbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 31 13:49:08.230696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:08.230662    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 31 13:49:08.230950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:08.230733    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9" podUID="4608de93-17e1-4803-a234-bfb4e24e559f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 31 13:49:08.514946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:08.496844    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:08.885961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:08.879486    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 31 13:49:08.885961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:08.879550    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb" podUID="fdbe827a-0f8e-41f0-a10e-0893d404efbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 31 13:49:09.231121 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:09.231080    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 31 13:49:09.231365 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:09.231147    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9" podUID="4608de93-17e1-4803-a234-bfb4e24e559f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 31 13:49:09.463893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:09.463083    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:09.879896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:09.876414    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 31 13:49:09.879896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:09.876498    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb" podUID="fdbe827a-0f8e-41f0-a10e-0893d404efbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 31 13:49:10.231158 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:10.231119    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 31 13:49:10.231597 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:10.231176    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9" podUID="4608de93-17e1-4803-a234-bfb4e24e559f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 31 13:49:10.463600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:10.463557    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:10.719883 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:10.719826    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:49:10.875896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:10.874993    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 31 13:49:10.875896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:10.875071    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb" podUID="fdbe827a-0f8e-41f0-a10e-0893d404efbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 31 13:49:11.136245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:11.136150    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:11.231278 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:11.231240    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 31 13:49:11.231711 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:11.231303    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9" podUID="4608de93-17e1-4803-a234-bfb4e24e559f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 31 13:49:11.404057 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:11.394730    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:11.874898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:11.874833    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 31 13:49:11.875924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:11.875886    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb" podUID="fdbe827a-0f8e-41f0-a10e-0893d404efbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 31 13:49:12.231672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:12.231638    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 31 13:49:12.232133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:12.231702    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9" podUID="4608de93-17e1-4803-a234-bfb4e24e559f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 31 13:49:12.297454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:12.297417    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:12.890896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:12.880698    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 31 13:49:12.890896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:12.880780    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb" podUID="fdbe827a-0f8e-41f0-a10e-0893d404efbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 31 13:49:13.234292 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:13.231271    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Oct 31 13:49:13.234292 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:13.231356    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9" podUID="4608de93-17e1-4803-a234-bfb4e24e559f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Oct 31 13:49:13.237448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:13.237413    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:14.028896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:14.028218    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:14.101537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:14.100885    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:49:14.102014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:14.101950    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:49:14.105318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:14.105291    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:49:14.236361 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:14.236331    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:49:14.237595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:14.237573    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 13:49:14.564242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:14.564196    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:14.929370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:14.929287    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:15.093909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:15.092769    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:15.413779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:15.413748    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:16.007894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:16.001251    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:17.022893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:17.022449    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:17.354699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:17.354611    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:17.995807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:17.990407    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:18.741957 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:49:18.741905    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:49:18.834939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:18.834585    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:19.610898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:19.608924    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:20.424891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:20.416226    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:20.938362 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:20.938330    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:21.431910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:21.427101    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:22.386556 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:22.386518    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:23.214804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:23.214755    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:24.087900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:24.083913    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:24.996894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:24.996574    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:25.814895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:25.812895    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:26.622352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:26.622309    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:27.525720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:27.525674    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:28.402088 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:28.402009    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:31.111015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:31.110955    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:31.731709 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:31.731671    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:32.600657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:32.600574    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:33.410765 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:33.410703    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:33.910806 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:33.910729    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:34.543182 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:34.543146    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:34.967954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:34.964951    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:35.548836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:35.548518    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:36.416445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:36.415493    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:36.747716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:36.747671    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:37.410883 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:37.410800    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:37.892261 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:37.892230    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:38.318522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:38.318483    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:39.092822 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:39.092784    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:39.862923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:39.862886    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:40.726897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:40.717730    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:41.508677 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:41.508640    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:42.296559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:42.294648    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:43.262820 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:43.262765    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:44.011588 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:44.011540    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:44.803775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:44.803741    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:45.665756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:45.665713    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:46.368128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:46.368087    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:46.780398 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:46.780356    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:47.609918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:47.608221    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:48.511171 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:48.510987    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:48.745249 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:49:48.745209    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:49:49.413579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:49.408185    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:49.833893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:49.832967    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:50.945945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:50.945905    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:52.181996 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:52.181956    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:53.130013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:53.129973    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:54.097715 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:54.097680    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:54.897572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:54.897515    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:55.804767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:55.804717    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:56.795261 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:56.795228    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:57.133099 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:57.133013    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:57.587936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:57.585749    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:58.386264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:58.386218    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:49:59.214525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:49:59.214467    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:00.040608 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:00.040567    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"]
Oct 31 13:50:00.089927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:00.089887    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:00.245557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:00.245524    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:01.091989 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:01.091950    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:01.944909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:01.936848    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:02.542322 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:02.542226    4428 generic.go:356] "Generic (PLEG): container finished" podID="5701fca5-38a3-4a28-86b3-55124cb6f54e" containerID="efe28536f4ea94d65dbf1603e4354ba9c1982b453fc165400e6ca54cec638912" exitCode=0
Oct 31 13:50:02.542322 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:02.542305    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh" event={"ID":"5701fca5-38a3-4a28-86b3-55124cb6f54e","Type":"ContainerDied","Data":"efe28536f4ea94d65dbf1603e4354ba9c1982b453fc165400e6ca54cec638912"}
Oct 31 13:50:02.542842 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:02.542790    4428 scope.go:122] "RemoveContainer" containerID="efe28536f4ea94d65dbf1603e4354ba9c1982b453fc165400e6ca54cec638912"
Oct 31 13:50:02.545240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:02.545205    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:50:02.545392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:02.545295    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:50:02.545392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:02.545343    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:50:02.811779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:02.808906    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:03.468895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:03.444846    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:03.781957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:03.781783    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh" event={"ID":"5701fca5-38a3-4a28-86b3-55124cb6f54e","Type":"ContainerStarted","Data":"c364208074403e6d29da90aa9d609acadada0a9da75616be12046e3d111f493c"}
Oct 31 13:50:03.880458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:03.880418    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:50:03.880681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:03.880652    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 13:50:03.880728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:03.880710    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh" podUID="5701fca5-38a3-4a28-86b3-55124cb6f54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 13:50:03.937898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:03.937402    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:04.880656 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:04.880615    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 13:50:04.881111 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:04.880709    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh" podUID="5701fca5-38a3-4a28-86b3-55124cb6f54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 13:50:04.989781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:04.989746    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:05.042643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:05.042571    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd" podUID="eb630d97-8786-4349-b029-96ed7eda31a5" containerName="payload-pre-processing" containerID="cri-o://814ab9606c651e4a556e2b516103aef26e2a999f1cc6c0d84d2a3b3b48a25e9f" gracePeriod=25
Oct 31 13:50:05.781049 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:05.781008    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:05.881494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:05.881455    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 13:50:05.882007 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:05.881537    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh" podUID="5701fca5-38a3-4a28-86b3-55124cb6f54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 13:50:06.694672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:06.694630    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:06.880615 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:06.880573    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 13:50:06.880823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:06.880643    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh" podUID="5701fca5-38a3-4a28-86b3-55124cb6f54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 13:50:07.617363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:07.617279    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:07.880623 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:07.880532    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 13:50:07.880623 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:07.880589    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh" podUID="5701fca5-38a3-4a28-86b3-55124cb6f54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 13:50:08.471998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:08.471964    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:08.881132 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:08.881039    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 13:50:08.881132 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:08.881099    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh" podUID="5701fca5-38a3-4a28-86b3-55124cb6f54e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 13:50:09.073285 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:09.073243    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:50:09.296176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:09.296108    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:09.887396 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:09.887362    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:50:09.888735 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:09.888704    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 13:50:10.118111 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:10.117065    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:10.886893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:10.885181    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:11.640521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:11.640469    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:12.009925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:12.009888    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd" podUID="eb630d97-8786-4349-b029-96ed7eda31a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:50:12.160070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:12.160030    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:12.317510 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:12.317424    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:12.473430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:12.473393    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:13.201452 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:13.201415    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:13.816992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:13.816954    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:14.007892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:14.007384    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:14.373305 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:14.373276    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:14.528813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:14.528773    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:14.685399 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:14.685308    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:14.859201 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:14.859166    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:15.384496 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:15.384457    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:15.545689 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:15.545639    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:16.203308 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:16.203259    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:16.960633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:16.960595    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:17.800881 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:17.800840    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:18.684110 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:18.684064    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:18.829812 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:50:18.829775    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:50:19.171280 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:19.171223    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:19.884782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:19.884739    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:20.528137 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:20.528094    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:21.406070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:21.406022    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:21.751228 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:21.751182    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:21.908203 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:21.908161    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:22.009747 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:22.009663    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd" podUID="eb630d97-8786-4349-b029-96ed7eda31a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:50:22.105039 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:22.104192    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:22.349231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:22.349085    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:23.458194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:23.458159    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:24.275241 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:24.275204    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:25.104211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:25.089028    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:25.197965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:25.197924    4428 generic.go:356] "Generic (PLEG): container finished" podID="eb630d97-8786-4349-b029-96ed7eda31a5" containerID="814ab9606c651e4a556e2b516103aef26e2a999f1cc6c0d84d2a3b3b48a25e9f" exitCode=0
Oct 31 13:50:25.198149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:25.198008    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd" event={"ID":"eb630d97-8786-4349-b029-96ed7eda31a5","Type":"ContainerDied","Data":"814ab9606c651e4a556e2b516103aef26e2a999f1cc6c0d84d2a3b3b48a25e9f"}
Oct 31 13:50:25.369209 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:25.369179    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"
Oct 31 13:50:25.510731 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:25.510697    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb630d97-8786-4349-b029-96ed7eda31a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb630d97-8786-4349-b029-96ed7eda31a5-plugins-config-volume\") pod \"eb630d97-8786-4349-b029-96ed7eda31a5\" (UID: \"eb630d97-8786-4349-b029-96ed7eda31a5\") "
Oct 31 13:50:25.510976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:25.510911    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb630d97-8786-4349-b029-96ed7eda31a5-kube-api-access-b4znz\" (UniqueName: \"kubernetes.io/projected/eb630d97-8786-4349-b029-96ed7eda31a5-kube-api-access-b4znz\") pod \"eb630d97-8786-4349-b029-96ed7eda31a5\" (UID: \"eb630d97-8786-4349-b029-96ed7eda31a5\") "
Oct 31 13:50:25.511103 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:25.511080    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb630d97-8786-4349-b029-96ed7eda31a5-plugins-config-volume" pod "eb630d97-8786-4349-b029-96ed7eda31a5" (UID: "eb630d97-8786-4349-b029-96ed7eda31a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:50:25.511507 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:25.511487    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb630d97-8786-4349-b029-96ed7eda31a5-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:25.513353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:25.513332    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb630d97-8786-4349-b029-96ed7eda31a5-kube-api-access-b4znz" pod "eb630d97-8786-4349-b029-96ed7eda31a5" (UID: "eb630d97-8786-4349-b029-96ed7eda31a5"). InnerVolumeSpecName "kube-api-access-b4znz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:50:25.613366 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:25.613272    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4znz\" (UniqueName: \"kubernetes.io/projected/eb630d97-8786-4349-b029-96ed7eda31a5-kube-api-access-b4znz\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:25.937973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:25.923593    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:26.208377 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:26.208291    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"
Oct 31 13:50:26.208911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:26.208301    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd" event={"ID":"eb630d97-8786-4349-b029-96ed7eda31a5","Type":"ContainerDied","Data":"b2dcffbbd2c2e18ebda87e70d983fb5deb1f4adbfaf0ae048da1273801327009"}
Oct 31 13:50:26.208911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:26.208434    4428 scope.go:122] "RemoveContainer" containerID="814ab9606c651e4a556e2b516103aef26e2a999f1cc6c0d84d2a3b3b48a25e9f"
Oct 31 13:50:26.235640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:26.235613    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"]
Oct 31 13:50:26.241490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:26.241467    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-554759-5f8879f7c9-chpfd"]
Oct 31 13:50:26.757919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:26.757879    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb630d97-8786-4349-b029-96ed7eda31a5" path="/var/lib/kubelet/pods/eb630d97-8786-4349-b029-96ed7eda31a5/volumes"
Oct 31 13:50:26.917488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:26.917438    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:27.030761 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:27.030656    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"]
Oct 31 13:50:27.036344 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:27.036299    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc2b72a-hxnsw"]
Oct 31 13:50:27.040928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:27.040840    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"]
Oct 31 13:50:27.045243 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:27.045207    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9945cb57-7dtx4"]
Oct 31 13:50:28.266896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:28.262324    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:28.957343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:28.956435    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e053af8-843e-4184-8a8d-770834fc5cf2" path="/var/lib/kubelet/pods/3e053af8-843e-4184-8a8d-770834fc5cf2/volumes"
Oct 31 13:50:28.961776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:28.961743    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d07c1925-fc71-432f-9251-1d22de15efd0" path="/var/lib/kubelet/pods/d07c1925-fc71-432f-9251-1d22de15efd0/volumes"
Oct 31 13:50:29.411941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:29.411897    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:30.320723 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:30.320610    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:31.323893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:31.323428    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:32.311815 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:32.311780    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:32.655945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:32.655774    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:32.818671 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:32.818436    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:33.306803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:33.306732    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:33.471522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:33.471446    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:34.018191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:34.017882    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:34.422133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:34.422042    4428 scope.go:122] "RemoveContainer" containerID="bf04cb843d2f5a6404f855500c246184f6fe4a4d1248ad41576714064e188ba6"
Oct 31 13:50:34.790418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:34.790375    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:35.628926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:35.625404    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:36.482358 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:36.475433    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:38.332789 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:38.332750    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:38.492103 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:38.492063    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:38.838893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:38.833904    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:39.030255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:39.030223    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"]
Oct 31 13:50:39.034596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:39.034565    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1585303-8lskb"]
Oct 31 13:50:39.573664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:39.572396    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:40.367836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:40.358124    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:40.758973 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:40.758940    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da62d96-c7bc-4502-b7a2-4ea716efbe2e" path="/var/lib/kubelet/pods/2da62d96-c7bc-4502-b7a2-4ea716efbe2e/volumes"
Oct 31 13:50:41.034791 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.034685    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"]
Oct 31 13:50:41.039567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.039539    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d28499f-qqsbz"]
Oct 31 13:50:41.302729 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.296502    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:41.696772 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.696679    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"]
Oct 31 13:50:41.697698 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.697672    4428 cpu_manager.go:414] "containerMap: removing container" podUID="eb630d97-8786-4349-b029-96ed7eda31a5" containerName="payload-pre-processing"
Oct 31 13:50:41.697928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.697698    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb630d97-8786-4349-b029-96ed7eda31a5" containerName="payload-pre-processing"
Oct 31 13:50:41.698009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.697932    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb630d97-8786-4349-b029-96ed7eda31a5" containerName="payload-pre-processing"
Oct 31 13:50:41.703771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.703750    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.710073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.710046    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-6a3533-openshift-default-dockercfg-79xkn\""
Oct 31 13:50:41.713905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.713882    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"]
Oct 31 13:50:41.773378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.773331    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-workload-certs\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.773860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.773511    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-podinfo\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.773860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.773554    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-workload-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.773860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.773715    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bthx4\" (UniqueName: \"kubernetes.io/projected/2db01e87-ab22-48aa-bed3-8b927d2a2162-kube-api-access-bthx4\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.774078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.773851    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-data\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.774078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.773952    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-token\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.774200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.774118    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-envoy\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.774200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.774165    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-credential-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.774292 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.774272    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db01e87-ab22-48aa-bed3-8b927d2a2162-istiod-ca-cert\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.875676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.875626    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-data\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.875920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.875692    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-token\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.875920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.875784    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-envoy\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.875920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.875817    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-credential-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.875920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.875851    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db01e87-ab22-48aa-bed3-8b927d2a2162-istiod-ca-cert\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.876195 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.875949    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-workload-certs\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.876195 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.876087    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-podinfo\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.876195 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.876124    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-workload-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.876195 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.876152    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-data\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.876403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.876235    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bthx4\" (UniqueName: \"kubernetes.io/projected/2db01e87-ab22-48aa-bed3-8b927d2a2162-kube-api-access-bthx4\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.876403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.876316    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-credential-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.876568 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.876541    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-workload-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.876743 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.876722    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db01e87-ab22-48aa-bed3-8b927d2a2162-istiod-ca-cert\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.877015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.876734    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-workload-certs\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.878286 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.878263    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-envoy\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.878819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.878797    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-podinfo\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.889758 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.889731    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-token\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:41.889927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:41.889908    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bthx4\" (UniqueName: \"kubernetes.io/projected/2db01e87-ab22-48aa-bed3-8b927d2a2162-kube-api-access-bthx4\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:42.024021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:42.023971    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:42.118899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:42.117427    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:42.395084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:42.395054    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"]
Oct 31 13:50:42.395687 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:50:42.395660    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db01e87_ab22_48aa_bed3_8b927d2a2162.slice/crio-9038216c9739971ac294698187d21a44665c08f2d3dfb981bc811b1a8c631f74 WatchSource:0}: Error finding container 9038216c9739971ac294698187d21a44665c08f2d3dfb981bc811b1a8c631f74: Status 404 returned error can't find the container with id 9038216c9739971ac294698187d21a44665c08f2d3dfb981bc811b1a8c631f74
Oct 31 13:50:42.399074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:42.399036    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:50:42.399175 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:42.399110    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:50:42.399175 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:42.399141    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:50:42.476939 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:42.476905    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4" event={"ID":"2db01e87-ab22-48aa-bed3-8b927d2a2162","Type":"ContainerStarted","Data":"9038216c9739971ac294698187d21a44665c08f2d3dfb981bc811b1a8c631f74"}
Oct 31 13:50:42.760533 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:42.760500    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a06d3d41-b562-4a1a-8dea-3fabee0fb40c" path="/var/lib/kubelet/pods/a06d3d41-b562-4a1a-8dea-3fabee0fb40c/volumes"
Oct 31 13:50:42.978902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:42.972907    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:43.489140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:43.489094    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4" event={"ID":"2db01e87-ab22-48aa-bed3-8b927d2a2162","Type":"ContainerStarted","Data":"e1cbccfe4abec796e5de1ddbb6f557171664543b4b882e604ce3965acb0d2c23"}
Oct 31 13:50:43.522535 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:43.522471    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4" podStartSLOduration=2.522448204 podStartE2EDuration="2.522448204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:50:43.519460276 +0000 UTC m=+2840.917808898" watchObservedRunningTime="2026-10-31 13:50:43.522448204 +0000 UTC m=+2840.920796897"
Oct 31 13:50:44.010896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:44.008125    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:44.026894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:44.024847    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:44.034897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:44.032780    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 31 13:50:44.034897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:44.032914    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4" podUID="2db01e87-ab22-48aa-bed3-8b927d2a2162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 31 13:50:44.057212 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:44.056468    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"]
Oct 31 13:50:44.067955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:44.064668    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5a3559-fbrl5"]
Oct 31 13:50:44.679107 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:44.679072    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:44.783400 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:44.783362    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cd48437-2d0c-485f-952d-e5116c9514da" path="/var/lib/kubelet/pods/5cd48437-2d0c-485f-952d-e5116c9514da/volumes"
Oct 31 13:50:44.906891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:44.902945    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:45.026897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:45.025315    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 31 13:50:45.026897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:45.025392    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4" podUID="2db01e87-ab22-48aa-bed3-8b927d2a2162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 31 13:50:45.077897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:45.076901    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:45.402971 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:45.401326    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:45.570699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:45.570664    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:46.025407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:46.025333    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 31 13:50:46.025407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:46.025439    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4" podUID="2db01e87-ab22-48aa-bed3-8b927d2a2162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 31 13:50:46.330941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:46.328566    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:47.025529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:47.025499    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 31 13:50:47.025724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:47.025564    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4" podUID="2db01e87-ab22-48aa-bed3-8b927d2a2162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 31 13:50:47.265932 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:47.264927    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:48.024494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:48.024458    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 31 13:50:48.024927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:48.024527    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4" podUID="2db01e87-ab22-48aa-bed3-8b927d2a2162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 31 13:50:48.189469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:48.189315    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:48.742579 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:50:48.742538    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:50:48.923893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:48.921566    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:49.024373 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:49.024282    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 31 13:50:49.024373 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:49.024347    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4" podUID="2db01e87-ab22-48aa-bed3-8b927d2a2162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 31 13:50:49.821051 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:49.816200    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:50.029712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:50.029682    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:50.030161 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:50.030121    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:50.031046 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:50.031030    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:50:50.705896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:50.705398    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:51.436963 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:51.436926    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:52.330684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:52.330605    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:52.696769 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:52.696655    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:52.859262 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:52.859229    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:52.964177 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:52.964146    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"]
Oct 31 13:50:52.964386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:52.964341    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6" podUID="028e0271-4e2d-48fd-851f-d5e82e774031" containerName="istio-proxy" containerID="cri-o://1831ad23966254404676845fa07450c9301732506db557e6c5e109b5598a5c3c" gracePeriod=30
Oct 31 13:50:53.081894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:53.081722    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:53.252327 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:53.249626    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:54.042796 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:54.042757    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:54.509806 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:54.509768    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:55.499844 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:55.499801    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:56.280264 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:56.280205    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:57.026927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:57.022388    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:57.992220 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:57.992186    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:58.267441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.267355    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:50:58.400354 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.400310    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/028e0271-4e2d-48fd-851f-d5e82e774031-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/028e0271-4e2d-48fd-851f-d5e82e774031-istio-podinfo\") pod \"028e0271-4e2d-48fd-851f-d5e82e774031\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") "
Oct 31 13:50:58.400564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.400373    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-credential-socket\") pod \"028e0271-4e2d-48fd-851f-d5e82e774031\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") "
Oct 31 13:50:58.400564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.400531    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-istio-data\") pod \"028e0271-4e2d-48fd-851f-d5e82e774031\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") "
Oct 31 13:50:58.400693 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.400585    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-istio-envoy\") pod \"028e0271-4e2d-48fd-851f-d5e82e774031\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") "
Oct 31 13:50:58.400693 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.400613    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/028e0271-4e2d-48fd-851f-d5e82e774031-kube-api-access-5cw5q\" (UniqueName: \"kubernetes.io/projected/028e0271-4e2d-48fd-851f-d5e82e774031-kube-api-access-5cw5q\") pod \"028e0271-4e2d-48fd-851f-d5e82e774031\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") "
Oct 31 13:50:58.400800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.400711    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-credential-socket" pod "028e0271-4e2d-48fd-851f-d5e82e774031" (UID: "028e0271-4e2d-48fd-851f-d5e82e774031"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:58.400800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.400728    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-workload-socket\") pod \"028e0271-4e2d-48fd-851f-d5e82e774031\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") "
Oct 31 13:50:58.400937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.400802    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/028e0271-4e2d-48fd-851f-d5e82e774031-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/028e0271-4e2d-48fd-851f-d5e82e774031-istiod-ca-cert\") pod \"028e0271-4e2d-48fd-851f-d5e82e774031\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") "
Oct 31 13:50:58.400937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.400833    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-istio-data" pod "028e0271-4e2d-48fd-851f-d5e82e774031" (UID: "028e0271-4e2d-48fd-851f-d5e82e774031"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:58.400937 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.400930    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/028e0271-4e2d-48fd-851f-d5e82e774031-istio-token\" (UniqueName: \"kubernetes.io/projected/028e0271-4e2d-48fd-851f-d5e82e774031-istio-token\") pod \"028e0271-4e2d-48fd-851f-d5e82e774031\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") "
Oct 31 13:50:58.401099 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.401033    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-workload-socket" pod "028e0271-4e2d-48fd-851f-d5e82e774031" (UID: "028e0271-4e2d-48fd-851f-d5e82e774031"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:58.401099 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.401071    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-workload-certs\") pod \"028e0271-4e2d-48fd-851f-d5e82e774031\" (UID: \"028e0271-4e2d-48fd-851f-d5e82e774031\") "
Oct 31 13:50:58.401207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.401186    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028e0271-4e2d-48fd-851f-d5e82e774031-istiod-ca-cert" pod "028e0271-4e2d-48fd-851f-d5e82e774031" (UID: "028e0271-4e2d-48fd-851f-d5e82e774031"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:50:58.401418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.401397    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-workload-certs" pod "028e0271-4e2d-48fd-851f-d5e82e774031" (UID: "028e0271-4e2d-48fd-851f-d5e82e774031"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:58.402055 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.402029    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:58.402183 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.402058    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:58.402183 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.402075    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:58.402183 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.402090    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:58.402183 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.402105    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/028e0271-4e2d-48fd-851f-d5e82e774031-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:58.402924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.402860    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/028e0271-4e2d-48fd-851f-d5e82e774031-istio-podinfo" pod "028e0271-4e2d-48fd-851f-d5e82e774031" (UID: "028e0271-4e2d-48fd-851f-d5e82e774031"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:50:58.403213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.403147    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-istio-envoy" pod "028e0271-4e2d-48fd-851f-d5e82e774031" (UID: "028e0271-4e2d-48fd-851f-d5e82e774031"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:50:58.403805 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.403778    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028e0271-4e2d-48fd-851f-d5e82e774031-kube-api-access-5cw5q" pod "028e0271-4e2d-48fd-851f-d5e82e774031" (UID: "028e0271-4e2d-48fd-851f-d5e82e774031"). InnerVolumeSpecName "kube-api-access-5cw5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:50:58.403969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.403807    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028e0271-4e2d-48fd-851f-d5e82e774031-istio-token" pod "028e0271-4e2d-48fd-851f-d5e82e774031" (UID: "028e0271-4e2d-48fd-851f-d5e82e774031"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:50:58.503475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.503430    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/028e0271-4e2d-48fd-851f-d5e82e774031-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:58.503475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.503466    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cw5q\" (UniqueName: \"kubernetes.io/projected/028e0271-4e2d-48fd-851f-d5e82e774031-kube-api-access-5cw5q\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:58.503475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.503477    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/028e0271-4e2d-48fd-851f-d5e82e774031-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:58.503475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.503486    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/028e0271-4e2d-48fd-851f-d5e82e774031-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:50:58.644366 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.644279    4428 generic.go:356] "Generic (PLEG): container finished" podID="028e0271-4e2d-48fd-851f-d5e82e774031" containerID="1831ad23966254404676845fa07450c9301732506db557e6c5e109b5598a5c3c" exitCode=0
Oct 31 13:50:58.644524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.644381    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"
Oct 31 13:50:58.644524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.644407    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6" event={"ID":"028e0271-4e2d-48fd-851f-d5e82e774031","Type":"ContainerDied","Data":"1831ad23966254404676845fa07450c9301732506db557e6c5e109b5598a5c3c"}
Oct 31 13:50:58.644524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.644487    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6" event={"ID":"028e0271-4e2d-48fd-851f-d5e82e774031","Type":"ContainerDied","Data":"7211025c75842926a4f6a5df424cec68d2207244ff7e3b09419baac11a23871c"}
Oct 31 13:50:58.644524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.644509    4428 scope.go:122] "RemoveContainer" containerID="1831ad23966254404676845fa07450c9301732506db557e6c5e109b5598a5c3c"
Oct 31 13:50:58.657795 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.657773    4428 scope.go:122] "RemoveContainer" containerID="1831ad23966254404676845fa07450c9301732506db557e6c5e109b5598a5c3c"
Oct 31 13:50:58.659135 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:50:58.659115    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1831ad23966254404676845fa07450c9301732506db557e6c5e109b5598a5c3c\": container with ID starting with 1831ad23966254404676845fa07450c9301732506db557e6c5e109b5598a5c3c not found: ID does not exist" containerID="1831ad23966254404676845fa07450c9301732506db557e6c5e109b5598a5c3c"
Oct 31 13:50:58.659222 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.659143    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1831ad23966254404676845fa07450c9301732506db557e6c5e109b5598a5c3c"} err="failed to get container status \"1831ad23966254404676845fa07450c9301732506db557e6c5e109b5598a5c3c\": rpc error: code = NotFound desc = could not find container \"1831ad23966254404676845fa07450c9301732506db557e6c5e109b5598a5c3c\": container with ID starting with 1831ad23966254404676845fa07450c9301732506db557e6c5e109b5598a5c3c not found: ID does not exist"
Oct 31 13:50:58.688902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.687925    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"]
Oct 31 13:50:58.710895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.708847    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:50:58.747965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.747927    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-35b584-openshift-default-556b575787-8lxl6"]
Oct 31 13:50:58.758176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:58.758147    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="028e0271-4e2d-48fd-851f-d5e82e774031" path="/var/lib/kubelet/pods/028e0271-4e2d-48fd-851f-d5e82e774031/volumes"
Oct 31 13:50:59.474897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:50:59.474836    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:00.313925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:00.299378    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:01.185908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.183343    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:01.320168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.320134    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"]
Oct 31 13:51:01.320993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.320978    4428 cpu_manager.go:414] "containerMap: removing container" podUID="028e0271-4e2d-48fd-851f-d5e82e774031" containerName="istio-proxy"
Oct 31 13:51:01.321056 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.320995    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="028e0271-4e2d-48fd-851f-d5e82e774031" containerName="istio-proxy"
Oct 31 13:51:01.321135 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.321126    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="028e0271-4e2d-48fd-851f-d5e82e774031" containerName="istio-proxy"
Oct 31 13:51:01.326500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.326482    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.332613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.332588    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-14ba5a-openshift-default-dockercfg-qj6n9\""
Oct 31 13:51:01.339148 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.339123    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"]
Oct 31 13:51:01.361922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.361854    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-token\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.362102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.362017    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-workload-certs\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.362102 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.362046    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm6wt\" (UniqueName: \"kubernetes.io/projected/c1dfb224-4987-4a65-9cd3-56a3383aac7f-kube-api-access-mm6wt\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.362266 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.362118    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-envoy\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.362266 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.362164    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-workload-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.362266 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.362210    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-credential-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.362448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.362312    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istiod-ca-cert\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.362448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.362338    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-podinfo\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.362570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.362539    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-data\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.464262 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.464234    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istiod-ca-cert\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.464434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.464282    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-podinfo\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.464434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.464408    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-data\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.464524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.464434    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-token\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.464583 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.464539    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-workload-certs\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.464583 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.464561    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm6wt\" (UniqueName: \"kubernetes.io/projected/c1dfb224-4987-4a65-9cd3-56a3383aac7f-kube-api-access-mm6wt\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.464688 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.464637    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-envoy\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.464745 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.464693    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-workload-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.464801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.464744    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-credential-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.464860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.464844    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-data\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.464994 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.464911    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-workload-certs\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.465094 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.465073    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istiod-ca-cert\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.465185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.465166    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-workload-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.465246 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.465205    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-credential-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.467089 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.467067    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-envoy\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.467265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.467246    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-podinfo\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.477636 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.477609    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-token\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.478319 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.478294    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm6wt\" (UniqueName: \"kubernetes.io/projected/c1dfb224-4987-4a65-9cd3-56a3383aac7f-kube-api-access-mm6wt\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.645805 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.645774    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:01.941657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.941621    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"]
Oct 31 13:51:01.941999 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:51:01.941961    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1dfb224_4987_4a65_9cd3_56a3383aac7f.slice/crio-a72662f2f2f69e261656c4a9fe3ea8dbe1ba61216ac404411fdb2ec5b65f393b WatchSource:0}: Error finding container a72662f2f2f69e261656c4a9fe3ea8dbe1ba61216ac404411fdb2ec5b65f393b: Status 404 returned error can't find the container with id a72662f2f2f69e261656c4a9fe3ea8dbe1ba61216ac404411fdb2ec5b65f393b
Oct 31 13:51:01.945342 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.945307    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:51:01.945470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.945384    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:51:01.945470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:01.945414    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:51:02.035982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:02.028033    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:02.680740 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:02.680703    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2" event={"ID":"c1dfb224-4987-4a65-9cd3-56a3383aac7f","Type":"ContainerStarted","Data":"8cc88ea0301d428bd54c91620f0c7e1e87b92176c21a9d8de5e9ef02862128e3"}
Oct 31 13:51:02.681187 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:02.680789    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2" event={"ID":"c1dfb224-4987-4a65-9cd3-56a3383aac7f","Type":"ContainerStarted","Data":"a72662f2f2f69e261656c4a9fe3ea8dbe1ba61216ac404411fdb2ec5b65f393b"}
Oct 31 13:51:02.713534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:02.713482    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2" podStartSLOduration=1.713467415 podStartE2EDuration="1.713467415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:51:02.709050901 +0000 UTC m=+2860.107399541" watchObservedRunningTime="2026-10-31 13:51:02.713467415 +0000 UTC m=+2860.111816040"
Oct 31 13:51:02.872891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:02.872453    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:03.653901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:03.650220    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:03.671258 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:03.671221    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 31 13:51:03.671456 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:03.671292    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2" podUID="c1dfb224-4987-4a65-9cd3-56a3383aac7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 31 13:51:03.888618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:03.888579    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:04.670894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:04.670494    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 31 13:51:04.670894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:04.670583    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2" podUID="c1dfb224-4987-4a65-9cd3-56a3383aac7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 31 13:51:04.739066 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:04.739028    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:05.611888 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:05.611110    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:05.674830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:05.674792    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 31 13:51:05.675174 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:05.675140    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2" podUID="c1dfb224-4987-4a65-9cd3-56a3383aac7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 31 13:51:06.507895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:06.505295    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:06.670968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:06.670237    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 31 13:51:06.670968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:06.670325    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2" podUID="c1dfb224-4987-4a65-9cd3-56a3383aac7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 31 13:51:07.472789 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:07.472748    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:07.654668 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:07.654632    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 31 13:51:07.660053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:07.660009    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2" podUID="c1dfb224-4987-4a65-9cd3-56a3383aac7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 31 13:51:08.315458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:08.315420    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:08.668952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:08.660115    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 31 13:51:08.668952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:08.660175    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2" podUID="c1dfb224-4987-4a65-9cd3-56a3383aac7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 31 13:51:09.232464 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:09.232429    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:09.652538 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:09.652457    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:09.652983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:09.652908    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:09.653823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:09.653806    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 13:51:10.128936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:10.125364    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:10.962724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:10.962680    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:11.819432 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:11.819351    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:12.164351 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:12.164213    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:12.702171 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:12.702109    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:13.657734 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:13.657685    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:13.960321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:13.960289    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:14.632345 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:14.632307    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:14.840568 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:14.840534    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:15.535788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:15.535754    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:15.692449 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:15.692419    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:16.582174 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:16.582115    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:17.461848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:17.461809    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:18.378518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:18.378474    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:18.743815 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:51:18.743775    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:51:19.204328 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:19.204230    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:19.977067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:19.977029    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:20.950891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:20.947203    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:22.177934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:22.173067    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:23.319238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:23.319175    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:24.557623 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:24.557583    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:25.793201 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:25.793157    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:26.716426 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:26.716383    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:27.517354 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:27.517318    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:28.351897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:28.344775    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:29.430901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:29.430334    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:29.758087 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:29.758052    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:30.584930 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:30.584847    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:31.490576 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:31.490538    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:31.661723 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:31.661681    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"]
Oct 31 13:51:31.697424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:31.697387    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:32.643710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:32.640574    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:33.497033 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:33.496993    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:34.485860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:34.485805    4428 scope.go:122] "RemoveContainer" containerID="ca144209c1fd51b2d8ad611ac02cae7e7a60eaa4bba568bcf087452130845b68"
Oct 31 13:51:34.524770 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:34.524729    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:34.548563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:34.548532    4428 scope.go:122] "RemoveContainer" containerID="8f953586f2f032636c9272c50c67dc6765c892ebeccfdb049ccb8070c9efb11a"
Oct 31 13:51:35.326908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:35.326855    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:36.137673 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:36.137634    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:37.055894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:37.055342    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:38.007951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:38.007831    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:38.986119 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:38.986070    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:39.836489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:39.836454    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:41.011761 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:41.011672    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:41.914185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:41.914149    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:41.947560 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:41.947527    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"]
Oct 31 13:51:42.083603 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:42.083573    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:42.839564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:42.839526    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:43.831841 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:43.831798    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:44.644913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:44.637538    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:45.513893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:45.512375    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:46.419968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:46.419933    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:46.688086 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:46.687974    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf" podUID="afe8d194-92d6-471f-8233-15ba5c15b33a" containerName="main" containerID="cri-o://07718137f5914979f96db589df23fa21d809a92ea9a6f793c2fbc8e436a7d8de" gracePeriod=45
Oct 31 13:51:46.947981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:46.947840    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7" podUID="42ac93e6-9ed8-4aee-b19d-3e55fbd4170f" containerName="payload-pre-processing" containerID="cri-o://867716c344290a0bd78a7b3e8bcbd8a5496eadecd397512c5dddbff2cc52c892" gracePeriod=25
Oct 31 13:51:46.995390 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:46.995357    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:51:47.042272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.042234    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe8d194-92d6-471f-8233-15ba5c15b33a-kube-api-access-cjztz\" (UniqueName: \"kubernetes.io/projected/afe8d194-92d6-471f-8233-15ba5c15b33a-kube-api-access-cjztz\") pod \"afe8d194-92d6-471f-8233-15ba5c15b33a\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") "
Oct 31 13:51:47.042504 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.042479    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-home\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-home\") pod \"afe8d194-92d6-471f-8233-15ba5c15b33a\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") "
Oct 31 13:51:47.042585 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.042523    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-tmp-dir\") pod \"afe8d194-92d6-471f-8233-15ba5c15b33a\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") "
Oct 31 13:51:47.042749 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.042723    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-model-cache\") pod \"afe8d194-92d6-471f-8233-15ba5c15b33a\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") "
Oct 31 13:51:47.042834 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.042779    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afe8d194-92d6-471f-8233-15ba5c15b33a-tls-certs\" (UniqueName: \"kubernetes.io/secret/afe8d194-92d6-471f-8233-15ba5c15b33a-tls-certs\") pod \"afe8d194-92d6-471f-8233-15ba5c15b33a\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") "
Oct 31 13:51:47.042834 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.042813    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-home" pod "afe8d194-92d6-471f-8233-15ba5c15b33a" (UID: "afe8d194-92d6-471f-8233-15ba5c15b33a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:51:47.043030 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.042826    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-dshm\") pod \"afe8d194-92d6-471f-8233-15ba5c15b33a\" (UID: \"afe8d194-92d6-471f-8233-15ba5c15b33a\") "
Oct 31 13:51:47.043341 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.043316    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-tmp-dir" pod "afe8d194-92d6-471f-8233-15ba5c15b33a" (UID: "afe8d194-92d6-471f-8233-15ba5c15b33a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:51:47.043697 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.043671    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-model-cache" pod "afe8d194-92d6-471f-8233-15ba5c15b33a" (UID: "afe8d194-92d6-471f-8233-15ba5c15b33a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:51:47.044030 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.044006    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:51:47.044191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.044175    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:51:47.044306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.044292    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:51:47.048074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.047904    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe8d194-92d6-471f-8233-15ba5c15b33a-tls-certs" pod "afe8d194-92d6-471f-8233-15ba5c15b33a" (UID: "afe8d194-92d6-471f-8233-15ba5c15b33a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:51:47.048074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.048028    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe8d194-92d6-471f-8233-15ba5c15b33a-kube-api-access-cjztz" pod "afe8d194-92d6-471f-8233-15ba5c15b33a" (UID: "afe8d194-92d6-471f-8233-15ba5c15b33a"). InnerVolumeSpecName "kube-api-access-cjztz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:51:47.060054 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.060019    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-dshm" pod "afe8d194-92d6-471f-8233-15ba5c15b33a" (UID: "afe8d194-92d6-471f-8233-15ba5c15b33a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:51:47.147899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.147439    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjztz\" (UniqueName: \"kubernetes.io/projected/afe8d194-92d6-471f-8233-15ba5c15b33a-kube-api-access-cjztz\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:51:47.147899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.147480    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe8d194-92d6-471f-8233-15ba5c15b33a-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:51:47.147899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.147497    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afe8d194-92d6-471f-8233-15ba5c15b33a-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:51:47.159331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.156063    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:47.346032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.346000    4428 generic.go:356] "Generic (PLEG): container finished" podID="afe8d194-92d6-471f-8233-15ba5c15b33a" containerID="07718137f5914979f96db589df23fa21d809a92ea9a6f793c2fbc8e436a7d8de" exitCode=0
Oct 31 13:51:47.346242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.346050    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf" event={"ID":"afe8d194-92d6-471f-8233-15ba5c15b33a","Type":"ContainerDied","Data":"07718137f5914979f96db589df23fa21d809a92ea9a6f793c2fbc8e436a7d8de"}
Oct 31 13:51:47.346242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.346054    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"
Oct 31 13:51:47.346242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.346077    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf" event={"ID":"afe8d194-92d6-471f-8233-15ba5c15b33a","Type":"ContainerDied","Data":"dc6008af4f35e9e57b069e63adc824b402eaab47a043b7dc3d8632eadd06d755"}
Oct 31 13:51:47.346242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.346094    4428 scope.go:122] "RemoveContainer" containerID="07718137f5914979f96db589df23fa21d809a92ea9a6f793c2fbc8e436a7d8de"
Oct 31 13:51:47.359061 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.359039    4428 scope.go:122] "RemoveContainer" containerID="07718137f5914979f96db589df23fa21d809a92ea9a6f793c2fbc8e436a7d8de"
Oct 31 13:51:47.359407 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:51:47.359380    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07718137f5914979f96db589df23fa21d809a92ea9a6f793c2fbc8e436a7d8de\": container with ID starting with 07718137f5914979f96db589df23fa21d809a92ea9a6f793c2fbc8e436a7d8de not found: ID does not exist" containerID="07718137f5914979f96db589df23fa21d809a92ea9a6f793c2fbc8e436a7d8de"
Oct 31 13:51:47.359468 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.359418    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07718137f5914979f96db589df23fa21d809a92ea9a6f793c2fbc8e436a7d8de"} err="failed to get container status \"07718137f5914979f96db589df23fa21d809a92ea9a6f793c2fbc8e436a7d8de\": rpc error: code = NotFound desc = could not find container \"07718137f5914979f96db589df23fa21d809a92ea9a6f793c2fbc8e436a7d8de\": container with ID starting with 07718137f5914979f96db589df23fa21d809a92ea9a6f793c2fbc8e436a7d8de not found: ID does not exist"
Oct 31 13:51:47.374837 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.374812    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"]
Oct 31 13:51:47.379694 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.379668    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-72c210/test-model-72c210-kserve-b4ffdc445-7bvxf"]
Oct 31 13:51:47.861706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.861663    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7" podUID="42ac93e6-9ed8-4aee-b19d-3e55fbd4170f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:51:47.943896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:47.937777    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:48.726838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:48.726798    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:48.789731 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:51:48.789692    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:51:48.790079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:48.790051    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe8d194-92d6-471f-8233-15ba5c15b33a" path="/var/lib/kubelet/pods/afe8d194-92d6-471f-8233-15ba5c15b33a/volumes"
Oct 31 13:51:49.783951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:49.781362    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:50.530896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:50.525435    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:51.625894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:51.624499    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:52.593143 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:52.593105    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:53.538901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:53.536341    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:54.514547 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:54.510912    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:55.608892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:55.608607    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:56.333268 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:56.333232    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:57.080891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:57.072902    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:57.861628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:57.861592    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7" podUID="42ac93e6-9ed8-4aee-b19d-3e55fbd4170f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:51:57.930898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:57.913583    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:58.900578 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:58.900499    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:59.546301 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:59.546251    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:51:59.747425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:51:59.747386    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:00.093442 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:00.093406    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:00.264921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:00.264184    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:00.635363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:00.635335    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:01.209911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:01.205885    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:02.004941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:02.004906    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:02.967679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:02.967615    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:03.932898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:03.920344    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:04.695992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:04.694797    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:05.626819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:05.626769    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:06.599398 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:06.599357    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:07.248441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.248410    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"
Oct 31 13:52:07.445524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.435439    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42ac93e6-9ed8-4aee-b19d-3e55fbd4170f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ac93e6-9ed8-4aee-b19d-3e55fbd4170f-plugins-config-volume\") pod \"42ac93e6-9ed8-4aee-b19d-3e55fbd4170f\" (UID: \"42ac93e6-9ed8-4aee-b19d-3e55fbd4170f\") "
Oct 31 13:52:07.446307 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.446266    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ac93e6-9ed8-4aee-b19d-3e55fbd4170f-kube-api-access-tjvgl\" (UniqueName: \"kubernetes.io/projected/42ac93e6-9ed8-4aee-b19d-3e55fbd4170f-kube-api-access-tjvgl\") pod \"42ac93e6-9ed8-4aee-b19d-3e55fbd4170f\" (UID: \"42ac93e6-9ed8-4aee-b19d-3e55fbd4170f\") "
Oct 31 13:52:07.467653 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.455233    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:07.467653 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.465527    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ac93e6-9ed8-4aee-b19d-3e55fbd4170f-plugins-config-volume" pod "42ac93e6-9ed8-4aee-b19d-3e55fbd4170f" (UID: "42ac93e6-9ed8-4aee-b19d-3e55fbd4170f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:52:07.468292 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.468236    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ac93e6-9ed8-4aee-b19d-3e55fbd4170f-kube-api-access-tjvgl" pod "42ac93e6-9ed8-4aee-b19d-3e55fbd4170f" (UID: "42ac93e6-9ed8-4aee-b19d-3e55fbd4170f"). InnerVolumeSpecName "kube-api-access-tjvgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:52:07.549119 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.549086    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ac93e6-9ed8-4aee-b19d-3e55fbd4170f-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:07.549119 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.549113    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjvgl\" (UniqueName: \"kubernetes.io/projected/42ac93e6-9ed8-4aee-b19d-3e55fbd4170f-kube-api-access-tjvgl\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:07.619576 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.619535    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:07.649395 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.649356    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"]
Oct 31 13:52:07.649684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.649656    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx" podUID="6349679a-67f7-41e5-b289-dc91302b5977" containerName="istio-proxy" containerID="cri-o://becc1a45e7795362df9130b5c399494a36ab0e5ced8c46a99b58f9e53257b719" gracePeriod=30
Oct 31 13:52:07.725076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.724972    4428 generic.go:356] "Generic (PLEG): container finished" podID="42ac93e6-9ed8-4aee-b19d-3e55fbd4170f" containerID="867716c344290a0bd78a7b3e8bcbd8a5496eadecd397512c5dddbff2cc52c892" exitCode=0
Oct 31 13:52:07.725076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.725027    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"
Oct 31 13:52:07.725076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.725027    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7" event={"ID":"42ac93e6-9ed8-4aee-b19d-3e55fbd4170f","Type":"ContainerDied","Data":"867716c344290a0bd78a7b3e8bcbd8a5496eadecd397512c5dddbff2cc52c892"}
Oct 31 13:52:07.725378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.725090    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7" event={"ID":"42ac93e6-9ed8-4aee-b19d-3e55fbd4170f","Type":"ContainerDied","Data":"5895abaddf85877753a0406820ad43ab79ebdea5bf1e860a011394f2e7f044ca"}
Oct 31 13:52:07.725378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.725122    4428 scope.go:122] "RemoveContainer" containerID="867716c344290a0bd78a7b3e8bcbd8a5496eadecd397512c5dddbff2cc52c892"
Oct 31 13:52:07.837735 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.837619    4428 scope.go:122] "RemoveContainer" containerID="867716c344290a0bd78a7b3e8bcbd8a5496eadecd397512c5dddbff2cc52c892"
Oct 31 13:52:07.860893 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:52:07.840294    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"867716c344290a0bd78a7b3e8bcbd8a5496eadecd397512c5dddbff2cc52c892\": container with ID starting with 867716c344290a0bd78a7b3e8bcbd8a5496eadecd397512c5dddbff2cc52c892 not found: ID does not exist" containerID="867716c344290a0bd78a7b3e8bcbd8a5496eadecd397512c5dddbff2cc52c892"
Oct 31 13:52:07.860893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.840342    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"867716c344290a0bd78a7b3e8bcbd8a5496eadecd397512c5dddbff2cc52c892"} err="failed to get container status \"867716c344290a0bd78a7b3e8bcbd8a5496eadecd397512c5dddbff2cc52c892\": rpc error: code = NotFound desc = could not find container \"867716c344290a0bd78a7b3e8bcbd8a5496eadecd397512c5dddbff2cc52c892\": container with ID starting with 867716c344290a0bd78a7b3e8bcbd8a5496eadecd397512c5dddbff2cc52c892 not found: ID does not exist"
Oct 31 13:52:07.892255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.882932    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"]
Oct 31 13:52:07.916625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.916584    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ddade-6b7b57b595-swkz7"]
Oct 31 13:52:07.926172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:07.916586    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:08.633649 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:08.623382    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:08.758146 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:08.758109    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ac93e6-9ed8-4aee-b19d-3e55fbd4170f" path="/var/lib/kubelet/pods/42ac93e6-9ed8-4aee-b19d-3e55fbd4170f/volumes"
Oct 31 13:52:09.652447 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:09.650437    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:10.386717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:10.386641    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:11.207385 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:11.207348    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:12.035891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:12.024239    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:12.793376 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:12.781562    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:12.831366 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:12.829851    4428 generic.go:356] "Generic (PLEG): container finished" podID="6349679a-67f7-41e5-b289-dc91302b5977" containerID="becc1a45e7795362df9130b5c399494a36ab0e5ced8c46a99b58f9e53257b719" exitCode=0
Oct 31 13:52:12.840443 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:12.840405    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx" event={"ID":"6349679a-67f7-41e5-b289-dc91302b5977","Type":"ContainerDied","Data":"becc1a45e7795362df9130b5c399494a36ab0e5ced8c46a99b58f9e53257b719"}
Oct 31 13:52:12.840766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:12.840740    4428 scope.go:122] "RemoveContainer" containerID="ef28b27b58d8d70c68447f988100a8adc7d58ec0c399ab24db584a2f8a845c7f"
Oct 31 13:52:13.030841 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.030814    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:52:13.089522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.089482    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-istio-envoy\") pod \"6349679a-67f7-41e5-b289-dc91302b5977\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") "
Oct 31 13:52:13.089730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.089552    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-workload-socket\") pod \"6349679a-67f7-41e5-b289-dc91302b5977\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") "
Oct 31 13:52:13.089730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.089663    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6349679a-67f7-41e5-b289-dc91302b5977-istio-token\" (UniqueName: \"kubernetes.io/projected/6349679a-67f7-41e5-b289-dc91302b5977-istio-token\") pod \"6349679a-67f7-41e5-b289-dc91302b5977\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") "
Oct 31 13:52:13.089884 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.089742    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6349679a-67f7-41e5-b289-dc91302b5977-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6349679a-67f7-41e5-b289-dc91302b5977-istio-podinfo\") pod \"6349679a-67f7-41e5-b289-dc91302b5977\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") "
Oct 31 13:52:13.089884 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.089804    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-credential-socket\") pod \"6349679a-67f7-41e5-b289-dc91302b5977\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") "
Oct 31 13:52:13.089997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.089920    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-workload-socket" pod "6349679a-67f7-41e5-b289-dc91302b5977" (UID: "6349679a-67f7-41e5-b289-dc91302b5977"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:52:13.089997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.089974    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6349679a-67f7-41e5-b289-dc91302b5977-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6349679a-67f7-41e5-b289-dc91302b5977-istiod-ca-cert\") pod \"6349679a-67f7-41e5-b289-dc91302b5977\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") "
Oct 31 13:52:13.090115 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.090022    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-workload-certs\") pod \"6349679a-67f7-41e5-b289-dc91302b5977\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") "
Oct 31 13:52:13.090115 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.090102    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6349679a-67f7-41e5-b289-dc91302b5977-kube-api-access-mwshd\" (UniqueName: \"kubernetes.io/projected/6349679a-67f7-41e5-b289-dc91302b5977-kube-api-access-mwshd\") pod \"6349679a-67f7-41e5-b289-dc91302b5977\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") "
Oct 31 13:52:13.090219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.090136    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-istio-data\") pod \"6349679a-67f7-41e5-b289-dc91302b5977\" (UID: \"6349679a-67f7-41e5-b289-dc91302b5977\") "
Oct 31 13:52:13.090707 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.090558    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-workload-certs" pod "6349679a-67f7-41e5-b289-dc91302b5977" (UID: "6349679a-67f7-41e5-b289-dc91302b5977"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:52:13.090707 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.090650    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-credential-socket" pod "6349679a-67f7-41e5-b289-dc91302b5977" (UID: "6349679a-67f7-41e5-b289-dc91302b5977"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:52:13.090904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.090859    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-istio-data" pod "6349679a-67f7-41e5-b289-dc91302b5977" (UID: "6349679a-67f7-41e5-b289-dc91302b5977"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:52:13.090978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.090955    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.091050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.090980    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.091050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.090996    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.091294 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.091273    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6349679a-67f7-41e5-b289-dc91302b5977-istiod-ca-cert" pod "6349679a-67f7-41e5-b289-dc91302b5977" (UID: "6349679a-67f7-41e5-b289-dc91302b5977"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:52:13.092791 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.092764    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6349679a-67f7-41e5-b289-dc91302b5977-istio-podinfo" pod "6349679a-67f7-41e5-b289-dc91302b5977" (UID: "6349679a-67f7-41e5-b289-dc91302b5977"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:52:13.093088 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.093063    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-istio-envoy" pod "6349679a-67f7-41e5-b289-dc91302b5977" (UID: "6349679a-67f7-41e5-b289-dc91302b5977"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:52:13.093335 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.093311    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6349679a-67f7-41e5-b289-dc91302b5977-istio-token" pod "6349679a-67f7-41e5-b289-dc91302b5977" (UID: "6349679a-67f7-41e5-b289-dc91302b5977"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:52:13.094382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.094348    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6349679a-67f7-41e5-b289-dc91302b5977-kube-api-access-mwshd" pod "6349679a-67f7-41e5-b289-dc91302b5977" (UID: "6349679a-67f7-41e5-b289-dc91302b5977"). InnerVolumeSpecName "kube-api-access-mwshd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:52:13.193234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.193195    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6349679a-67f7-41e5-b289-dc91302b5977-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.193234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.193225    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6349679a-67f7-41e5-b289-dc91302b5977-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.193234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.193235    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwshd\" (UniqueName: \"kubernetes.io/projected/6349679a-67f7-41e5-b289-dc91302b5977-kube-api-access-mwshd\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.193534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.193245    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.193534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.193254    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6349679a-67f7-41e5-b289-dc91302b5977-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.193534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.193287    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6349679a-67f7-41e5-b289-dc91302b5977-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:13.838804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.838775    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"
Oct 31 13:52:13.839320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.838810    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx" event={"ID":"6349679a-67f7-41e5-b289-dc91302b5977","Type":"ContainerDied","Data":"edb13254c9d78a03be65316926620419999b08a8e3a7a24b8a70841884c6e0e8"}
Oct 31 13:52:13.839320 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.838913    4428 scope.go:122] "RemoveContainer" containerID="becc1a45e7795362df9130b5c399494a36ab0e5ced8c46a99b58f9e53257b719"
Oct 31 13:52:13.869834 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.869804    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"]
Oct 31 13:52:13.875114 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:13.875088    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1ddade-openshift-default-5df978b75c-jndsx"]
Oct 31 13:52:14.758415 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:14.758375    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6349679a-67f7-41e5-b289-dc91302b5977" path="/var/lib/kubelet/pods/6349679a-67f7-41e5-b289-dc91302b5977/volumes"
Oct 31 13:52:14.809956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:14.809919    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:14.966337 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:14.966302    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:15.305074 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:15.304917    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:15.585303 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:15.585207    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:16.041169 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:16.041130    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:16.828893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:16.825732    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:17.661908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:17.657130    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:18.204804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:18.204767    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"]
Oct 31 13:52:18.225127 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:18.225073    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"]
Oct 31 13:52:18.336377 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:18.336317    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:18.499488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:18.499446    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:18.752347 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:52:18.752252    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:52:19.231860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:19.231810    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:20.026889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:20.024054    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:20.816975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:20.808887    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:21.542567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:21.527341    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:22.394892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:22.393163    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:23.206057 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:23.206011    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs" podUID="803c3db4-5311-4a96-8474-64304a5995ba" containerName="payload-processing" containerID="cri-o://08b0b8a391e9cace2c0e343492a58fa67eadacbcda35099398054aa402e1ec05" gracePeriod=25
Oct 31 13:52:23.226321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:23.226254    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2" podUID="1821cf0c-d453-4c5c-83bd-7c3e22666953" containerName="payload-pre-processing" containerID="cri-o://f27596d445e50248b653b6f314d8a86ceea2c0c4896d14253ac5dc4dfca5f2fa" gracePeriod=25
Oct 31 13:52:23.444748 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:23.444714    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:24.069047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:24.069008    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs" podUID="803c3db4-5311-4a96-8474-64304a5995ba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:52:24.236893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:24.231658    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:24.800513 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:24.800477    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"]
Oct 31 13:52:24.801035 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:24.801000    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp" podUID="e1a979ed-3892-4eb5-879b-956d93393cbf" containerName="maas-api" containerID="cri-o://74309bb702e4963146ef98682671569f6a564cd34a7fb71b0a08c9097efbbdfc" gracePeriod=30
Oct 31 13:52:24.900221 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:24.900187    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:24.931918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:24.931881    4428 generic.go:356] "Generic (PLEG): container finished" podID="e1a979ed-3892-4eb5-879b-956d93393cbf" containerID="74309bb702e4963146ef98682671569f6a564cd34a7fb71b0a08c9097efbbdfc" exitCode=0
Oct 31 13:52:24.931918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:24.931901    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp" event={"ID":"e1a979ed-3892-4eb5-879b-956d93393cbf","Type":"ContainerDied","Data":"74309bb702e4963146ef98682671569f6a564cd34a7fb71b0a08c9097efbbdfc"}
Oct 31 13:52:25.144564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.144117    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2" podUID="1821cf0c-d453-4c5c-83bd-7c3e22666953" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:52:25.194497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.194336    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:25.312971 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.312922    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:52:25.406912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.406785    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1a979ed-3892-4eb5-879b-956d93393cbf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1a979ed-3892-4eb5-879b-956d93393cbf-metrics-certs\") pod \"e1a979ed-3892-4eb5-879b-956d93393cbf\" (UID: \"e1a979ed-3892-4eb5-879b-956d93393cbf\") "
Oct 31 13:52:25.407098 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.406915    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a979ed-3892-4eb5-879b-956d93393cbf-kube-api-access-6lpqv\" (UniqueName: \"kubernetes.io/projected/e1a979ed-3892-4eb5-879b-956d93393cbf-kube-api-access-6lpqv\") pod \"e1a979ed-3892-4eb5-879b-956d93393cbf\" (UID: \"e1a979ed-3892-4eb5-879b-956d93393cbf\") "
Oct 31 13:52:25.407098 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.407049    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a979ed-3892-4eb5-879b-956d93393cbf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a979ed-3892-4eb5-879b-956d93393cbf-openshift-service-ca\") pod \"e1a979ed-3892-4eb5-879b-956d93393cbf\" (UID: \"e1a979ed-3892-4eb5-879b-956d93393cbf\") "
Oct 31 13:52:25.407246 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.407215    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1a979ed-3892-4eb5-879b-956d93393cbf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1a979ed-3892-4eb5-879b-956d93393cbf-maas-api-tls\") pod \"e1a979ed-3892-4eb5-879b-956d93393cbf\" (UID: \"e1a979ed-3892-4eb5-879b-956d93393cbf\") "
Oct 31 13:52:25.407427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.407398    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a979ed-3892-4eb5-879b-956d93393cbf-openshift-service-ca" pod "e1a979ed-3892-4eb5-879b-956d93393cbf" (UID: "e1a979ed-3892-4eb5-879b-956d93393cbf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:52:25.408293 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.408259    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a979ed-3892-4eb5-879b-956d93393cbf-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:25.409732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.409688    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1a979ed-3892-4eb5-879b-956d93393cbf-maas-api-tls" pod "e1a979ed-3892-4eb5-879b-956d93393cbf" (UID: "e1a979ed-3892-4eb5-879b-956d93393cbf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:52:25.409861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.409828    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1a979ed-3892-4eb5-879b-956d93393cbf-metrics-certs" pod "e1a979ed-3892-4eb5-879b-956d93393cbf" (UID: "e1a979ed-3892-4eb5-879b-956d93393cbf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:52:25.410011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.409991    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a979ed-3892-4eb5-879b-956d93393cbf-kube-api-access-6lpqv" pod "e1a979ed-3892-4eb5-879b-956d93393cbf" (UID: "e1a979ed-3892-4eb5-879b-956d93393cbf"). InnerVolumeSpecName "kube-api-access-6lpqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:52:25.480934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.480745    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:25.510960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.510787    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1a979ed-3892-4eb5-879b-956d93393cbf-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:25.510960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.510833    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lpqv\" (UniqueName: \"kubernetes.io/projected/e1a979ed-3892-4eb5-879b-956d93393cbf-kube-api-access-6lpqv\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:25.510960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.510848    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1a979ed-3892-4eb5-879b-956d93393cbf-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:25.652849 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.652710    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:25.941744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.941714    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"
Oct 31 13:52:25.941997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.941749    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp" event={"ID":"e1a979ed-3892-4eb5-879b-956d93393cbf","Type":"ContainerDied","Data":"515ab74c478be8edb1ebf62fe9dcda5f7d825ef30846c865a459275fd4e1d3cc"}
Oct 31 13:52:25.941997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.941831    4428 scope.go:122] "RemoveContainer" containerID="74309bb702e4963146ef98682671569f6a564cd34a7fb71b0a08c9097efbbdfc"
Oct 31 13:52:25.969304 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.969268    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"]
Oct 31 13:52:25.974551 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:25.974487    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72c210-78bc4d7c46-2k4gp"]
Oct 31 13:52:26.035126 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:26.035090    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:26.625019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:26.624978    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:26.758484 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:26.758449    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a979ed-3892-4eb5-879b-956d93393cbf" path="/var/lib/kubelet/pods/e1a979ed-3892-4eb5-879b-956d93393cbf/volumes"
Oct 31 13:52:27.579544 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:27.579500    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:28.443434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:28.443397    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:29.137894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:29.136007    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:29.910506 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:29.910471    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:30.687829 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:30.687733    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:31.526647 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:31.526356    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:32.313589 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:32.313556    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:33.178376 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:33.178345    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:33.351792 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:33.351742    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:33.526275 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:33.526227    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:34.025565 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.025509    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:34.081783 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.081742    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs" podUID="803c3db4-5311-4a96-8474-64304a5995ba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:52:34.506979 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.506940    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:34.605382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.605349    4428 scope.go:122] "RemoveContainer" containerID="4902afdd2cfbe96d232b61d5e453eb846ee24e8aedc8b4b370f53b008845977e"
Oct 31 13:52:34.616848 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.616824    4428 scope.go:122] "RemoveContainer" containerID="360603f0d465af1822bb2f1f9600feb44afe4c096cb0a17a88d64cb7676f8de0"
Oct 31 13:52:34.629001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.628974    4428 scope.go:122] "RemoveContainer" containerID="2e3d33c968d056f4bcc1588cca3df926b9483583d0884a73cc32588f73026028"
Oct 31 13:52:34.640306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.640287    4428 scope.go:122] "RemoveContainer" containerID="fd75ebb9237485312da43ce357e10a4a8f559227d30cd80145ce663149634252"
Oct 31 13:52:34.650265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.650242    4428 scope.go:122] "RemoveContainer" containerID="c6700bc7f1a353e6cefc34785d2baddcb864a323ef2a062daee3b9debc6d9ddd"
Oct 31 13:52:34.804604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.804518    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:34.872140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.872093    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"]
Oct 31 13:52:34.873449 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.873421    4428 cpu_manager.go:414] "containerMap: removing container" podUID="afe8d194-92d6-471f-8233-15ba5c15b33a" containerName="main"
Oct 31 13:52:34.873449 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.873447    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe8d194-92d6-471f-8233-15ba5c15b33a" containerName="main"
Oct 31 13:52:34.873672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.873469    4428 cpu_manager.go:414] "containerMap: removing container" podUID="42ac93e6-9ed8-4aee-b19d-3e55fbd4170f" containerName="payload-pre-processing"
Oct 31 13:52:34.873672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.873477    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ac93e6-9ed8-4aee-b19d-3e55fbd4170f" containerName="payload-pre-processing"
Oct 31 13:52:34.873672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.873501    4428 cpu_manager.go:414] "containerMap: removing container" podUID="6349679a-67f7-41e5-b289-dc91302b5977" containerName="istio-proxy"
Oct 31 13:52:34.873672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.873510    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6349679a-67f7-41e5-b289-dc91302b5977" containerName="istio-proxy"
Oct 31 13:52:34.873672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.873549    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e1a979ed-3892-4eb5-879b-956d93393cbf" containerName="maas-api"
Oct 31 13:52:34.873672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.873557    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a979ed-3892-4eb5-879b-956d93393cbf" containerName="maas-api"
Oct 31 13:52:34.874037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.873773    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a979ed-3892-4eb5-879b-956d93393cbf" containerName="maas-api"
Oct 31 13:52:34.874037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.873796    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6349679a-67f7-41e5-b289-dc91302b5977" containerName="istio-proxy"
Oct 31 13:52:34.874037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.873811    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe8d194-92d6-471f-8233-15ba5c15b33a" containerName="main"
Oct 31 13:52:34.874037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.873820    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ac93e6-9ed8-4aee-b19d-3e55fbd4170f" containerName="payload-pre-processing"
Oct 31 13:52:34.888169 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.883008    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:34.891226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.891181    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"]
Oct 31 13:52:34.891514 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.891332    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-20b8a4-openshift-default-dockercfg-tzb4f\""
Oct 31 13:52:34.975035 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:34.975000    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:35.031100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.029250    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-workload-socket\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.031100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.029344    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27f0595-0a45-4d55-a097-9c94398bd559-istiod-ca-cert\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.031100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.029497    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp78t\" (UniqueName: \"kubernetes.io/projected/c27f0595-0a45-4d55-a097-9c94398bd559-kube-api-access-rp78t\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.031100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.029573    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-istio-data\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.031100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.029624    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-workload-certs\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.031100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.029656    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-credential-socket\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.031100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.029708    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27f0595-0a45-4d55-a097-9c94398bd559-istio-token\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.031100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.029802    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-istio-envoy\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.031100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.029904    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27f0595-0a45-4d55-a097-9c94398bd559-istio-podinfo\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.151948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.148281    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-istio-envoy\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.151948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.148393    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27f0595-0a45-4d55-a097-9c94398bd559-istio-podinfo\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.151948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.148460    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-workload-socket\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.151948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.148527    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27f0595-0a45-4d55-a097-9c94398bd559-istiod-ca-cert\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.151948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.148678    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp78t\" (UniqueName: \"kubernetes.io/projected/c27f0595-0a45-4d55-a097-9c94398bd559-kube-api-access-rp78t\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.151948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.148743    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-istio-data\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.151948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.148793    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-workload-certs\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.151948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.148827    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-credential-socket\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.151948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.148898    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27f0595-0a45-4d55-a097-9c94398bd559-istio-token\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.178920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.176904    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-workload-certs\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.178920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.177355    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-istio-data\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.183900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.182440    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-istio-envoy\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.183900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.182720    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-credential-socket\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.202895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.200956    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-workload-socket\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.202895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.201710    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27f0595-0a45-4d55-a097-9c94398bd559-istiod-ca-cert\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.218889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.211365    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2" podUID="1821cf0c-d453-4c5c-83bd-7c3e22666953" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:52:35.218889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.214344    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:35.235582 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.234675    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27f0595-0a45-4d55-a097-9c94398bd559-istio-podinfo\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.235582 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.235162    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27f0595-0a45-4d55-a097-9c94398bd559-istio-token\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.235851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.235660    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp78t\" (UniqueName: \"kubernetes.io/projected/c27f0595-0a45-4d55-a097-9c94398bd559-kube-api-access-rp78t\") pod \"e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") " pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.509759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.509726    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:35.793947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.792983    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:35.862421 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:52:35.862389    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27f0595_0a45_4d55_a097_9c94398bd559.slice/crio-9e583526c3806bed5dbc580a6f7e2c3e22ffe14d6d9969c4ecab76f706739a0a WatchSource:0}: Error finding container 9e583526c3806bed5dbc580a6f7e2c3e22ffe14d6d9969c4ecab76f706739a0a: Status 404 returned error can't find the container with id 9e583526c3806bed5dbc580a6f7e2c3e22ffe14d6d9969c4ecab76f706739a0a
Oct 31 13:52:35.862604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.862580    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"]
Oct 31 13:52:35.865778 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.865745    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:52:35.865905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.865811    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:52:35.865905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:35.865843    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:52:36.134130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:36.134022    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj" event={"ID":"c27f0595-0a45-4d55-a097-9c94398bd559","Type":"ContainerStarted","Data":"0de7216dfc4aadace9a35e1c2f5d8cc98804c5dd1671d9a3ebb28260f3e7a142"}
Oct 31 13:52:36.134130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:36.134091    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj" event={"ID":"c27f0595-0a45-4d55-a097-9c94398bd559","Type":"ContainerStarted","Data":"9e583526c3806bed5dbc580a6f7e2c3e22ffe14d6d9969c4ecab76f706739a0a"}
Oct 31 13:52:36.169295 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:36.169237    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj" podStartSLOduration=2.169220899 podStartE2EDuration="2.169220899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:52:36.164341248 +0000 UTC m=+2953.562689883" watchObservedRunningTime="2026-10-31 13:52:36.169220899 +0000 UTC m=+2953.567569522"
Oct 31 13:52:36.510641 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:36.510595    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:36.517367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:36.517343    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:36.720494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:36.718952    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:37.142263 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:37.142231    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:37.143199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:37.143180    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:52:37.787894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:37.780782    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:38.604526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:38.604490    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:38.791170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:38.791134    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:39.728726 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:39.728687    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:39.925722 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:39.925688    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:40.220955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:40.210840    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:40.720959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:40.719067    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:41.447394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:41.440482    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:42.328551 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:42.327263    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:43.155701 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:43.155666    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:43.542674 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:43.542317    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"
Oct 31 13:52:43.579106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:43.578851    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"
Oct 31 13:52:43.607714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:43.607618    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803c3db4-5311-4a96-8474-64304a5995ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803c3db4-5311-4a96-8474-64304a5995ba-plugins-config-volume\") pod \"803c3db4-5311-4a96-8474-64304a5995ba\" (UID: \"803c3db4-5311-4a96-8474-64304a5995ba\") "
Oct 31 13:52:43.607714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:43.607674    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1821cf0c-d453-4c5c-83bd-7c3e22666953-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1821cf0c-d453-4c5c-83bd-7c3e22666953-plugins-config-volume\") pod \"1821cf0c-d453-4c5c-83bd-7c3e22666953\" (UID: \"1821cf0c-d453-4c5c-83bd-7c3e22666953\") "
Oct 31 13:52:43.607714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:43.607719    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1821cf0c-d453-4c5c-83bd-7c3e22666953-kube-api-access-5kjsk\" (UniqueName: \"kubernetes.io/projected/1821cf0c-d453-4c5c-83bd-7c3e22666953-kube-api-access-5kjsk\") pod \"1821cf0c-d453-4c5c-83bd-7c3e22666953\" (UID: \"1821cf0c-d453-4c5c-83bd-7c3e22666953\") "
Oct 31 13:52:43.608085 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:43.607818    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803c3db4-5311-4a96-8474-64304a5995ba-kube-api-access-wj9xr\" (UniqueName: \"kubernetes.io/projected/803c3db4-5311-4a96-8474-64304a5995ba-kube-api-access-wj9xr\") pod \"803c3db4-5311-4a96-8474-64304a5995ba\" (UID: \"803c3db4-5311-4a96-8474-64304a5995ba\") "
Oct 31 13:52:43.608338 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:43.608308    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1821cf0c-d453-4c5c-83bd-7c3e22666953-plugins-config-volume" pod "1821cf0c-d453-4c5c-83bd-7c3e22666953" (UID: "1821cf0c-d453-4c5c-83bd-7c3e22666953"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:52:43.608604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:43.608578    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1821cf0c-d453-4c5c-83bd-7c3e22666953-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:43.608912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:43.608800    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803c3db4-5311-4a96-8474-64304a5995ba-plugins-config-volume" pod "803c3db4-5311-4a96-8474-64304a5995ba" (UID: "803c3db4-5311-4a96-8474-64304a5995ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:52:43.611141 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:43.611116    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1821cf0c-d453-4c5c-83bd-7c3e22666953-kube-api-access-5kjsk" pod "1821cf0c-d453-4c5c-83bd-7c3e22666953" (UID: "1821cf0c-d453-4c5c-83bd-7c3e22666953"). InnerVolumeSpecName "kube-api-access-5kjsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:52:43.611251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:43.611221    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803c3db4-5311-4a96-8474-64304a5995ba-kube-api-access-wj9xr" pod "803c3db4-5311-4a96-8474-64304a5995ba" (UID: "803c3db4-5311-4a96-8474-64304a5995ba"). InnerVolumeSpecName "kube-api-access-wj9xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:52:43.710331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:43.710299    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803c3db4-5311-4a96-8474-64304a5995ba-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:43.710331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:43.710332    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kjsk\" (UniqueName: \"kubernetes.io/projected/1821cf0c-d453-4c5c-83bd-7c3e22666953-kube-api-access-5kjsk\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:43.710543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:43.710343    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj9xr\" (UniqueName: \"kubernetes.io/projected/803c3db4-5311-4a96-8474-64304a5995ba-kube-api-access-wj9xr\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:52:43.914898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:43.906291    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:44.282667 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.282629    4428 generic.go:356] "Generic (PLEG): container finished" podID="1821cf0c-d453-4c5c-83bd-7c3e22666953" containerID="f27596d445e50248b653b6f314d8a86ceea2c0c4896d14253ac5dc4dfca5f2fa" exitCode=0
Oct 31 13:52:44.283367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.282691    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"
Oct 31 13:52:44.283367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.282714    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2" event={"ID":"1821cf0c-d453-4c5c-83bd-7c3e22666953","Type":"ContainerDied","Data":"f27596d445e50248b653b6f314d8a86ceea2c0c4896d14253ac5dc4dfca5f2fa"}
Oct 31 13:52:44.283367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.282772    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2" event={"ID":"1821cf0c-d453-4c5c-83bd-7c3e22666953","Type":"ContainerDied","Data":"a775b1a12b81d7be3ee631c56df79732feabbad02eed7f23fdcc4a76d23e83ac"}
Oct 31 13:52:44.283367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.282801    4428 scope.go:122] "RemoveContainer" containerID="f27596d445e50248b653b6f314d8a86ceea2c0c4896d14253ac5dc4dfca5f2fa"
Oct 31 13:52:44.285278 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.285249    4428 generic.go:356] "Generic (PLEG): container finished" podID="803c3db4-5311-4a96-8474-64304a5995ba" containerID="08b0b8a391e9cace2c0e343492a58fa67eadacbcda35099398054aa402e1ec05" exitCode=0
Oct 31 13:52:44.285411 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.285281    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs" event={"ID":"803c3db4-5311-4a96-8474-64304a5995ba","Type":"ContainerDied","Data":"08b0b8a391e9cace2c0e343492a58fa67eadacbcda35099398054aa402e1ec05"}
Oct 31 13:52:44.285411 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.285318    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"
Oct 31 13:52:44.285411 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.285341    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs" event={"ID":"803c3db4-5311-4a96-8474-64304a5995ba","Type":"ContainerDied","Data":"90934be146edb9eb4cd191c4f0bf07d7354adc146e1affebc4669ce507624d09"}
Oct 31 13:52:44.297515 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.297491    4428 scope.go:122] "RemoveContainer" containerID="f27596d445e50248b653b6f314d8a86ceea2c0c4896d14253ac5dc4dfca5f2fa"
Oct 31 13:52:44.297830 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:52:44.297810    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27596d445e50248b653b6f314d8a86ceea2c0c4896d14253ac5dc4dfca5f2fa\": container with ID starting with f27596d445e50248b653b6f314d8a86ceea2c0c4896d14253ac5dc4dfca5f2fa not found: ID does not exist" containerID="f27596d445e50248b653b6f314d8a86ceea2c0c4896d14253ac5dc4dfca5f2fa"
Oct 31 13:52:44.297907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.297839    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27596d445e50248b653b6f314d8a86ceea2c0c4896d14253ac5dc4dfca5f2fa"} err="failed to get container status \"f27596d445e50248b653b6f314d8a86ceea2c0c4896d14253ac5dc4dfca5f2fa\": rpc error: code = NotFound desc = could not find container \"f27596d445e50248b653b6f314d8a86ceea2c0c4896d14253ac5dc4dfca5f2fa\": container with ID starting with f27596d445e50248b653b6f314d8a86ceea2c0c4896d14253ac5dc4dfca5f2fa not found: ID does not exist"
Oct 31 13:52:44.297907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.297858    4428 scope.go:122] "RemoveContainer" containerID="08b0b8a391e9cace2c0e343492a58fa67eadacbcda35099398054aa402e1ec05"
Oct 31 13:52:44.310890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.310856    4428 scope.go:122] "RemoveContainer" containerID="08b0b8a391e9cace2c0e343492a58fa67eadacbcda35099398054aa402e1ec05"
Oct 31 13:52:44.311228 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:52:44.311208    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b0b8a391e9cace2c0e343492a58fa67eadacbcda35099398054aa402e1ec05\": container with ID starting with 08b0b8a391e9cace2c0e343492a58fa67eadacbcda35099398054aa402e1ec05 not found: ID does not exist" containerID="08b0b8a391e9cace2c0e343492a58fa67eadacbcda35099398054aa402e1ec05"
Oct 31 13:52:44.311282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.311235    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b0b8a391e9cace2c0e343492a58fa67eadacbcda35099398054aa402e1ec05"} err="failed to get container status \"08b0b8a391e9cace2c0e343492a58fa67eadacbcda35099398054aa402e1ec05\": rpc error: code = NotFound desc = could not find container \"08b0b8a391e9cace2c0e343492a58fa67eadacbcda35099398054aa402e1ec05\": container with ID starting with 08b0b8a391e9cace2c0e343492a58fa67eadacbcda35099398054aa402e1ec05 not found: ID does not exist"
Oct 31 13:52:44.312629 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.312608    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"]
Oct 31 13:52:44.323356 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.323329    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-72c210-5f66dbcb59-hmtc2"]
Oct 31 13:52:44.336454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.336426    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"]
Oct 31 13:52:44.342148 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.342124    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-72c210-68bd768b5-tt4vs"]
Oct 31 13:52:44.747893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.732998    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:44.762629 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.762592    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1821cf0c-d453-4c5c-83bd-7c3e22666953" path="/var/lib/kubelet/pods/1821cf0c-d453-4c5c-83bd-7c3e22666953/volumes"
Oct 31 13:52:44.763224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:44.763187    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="803c3db4-5311-4a96-8474-64304a5995ba" path="/var/lib/kubelet/pods/803c3db4-5311-4a96-8474-64304a5995ba/volumes"
Oct 31 13:52:45.507893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:45.497479    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:46.268986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:46.268945    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:47.116767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:47.116730    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:47.991185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:47.991048    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:48.742898 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:52:48.742846    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:52:48.862640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:48.862582    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:49.602898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:49.599152    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:50.425378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:50.425342    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:51.226614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:51.224513    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:52.000925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:51.999177    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:52.877497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:52.877455    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:54.094703 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:54.094633    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:55.092281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:55.087753    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:56.000813 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.000775    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:56.798139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.798103    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"]
Oct 31 13:52:56.799241 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.799219    4428 cpu_manager.go:414] "containerMap: removing container" podUID="803c3db4-5311-4a96-8474-64304a5995ba" containerName="payload-processing"
Oct 31 13:52:56.799241 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.799241    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803c3db4-5311-4a96-8474-64304a5995ba" containerName="payload-processing"
Oct 31 13:52:56.799378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.799258    4428 cpu_manager.go:414] "containerMap: removing container" podUID="6349679a-67f7-41e5-b289-dc91302b5977" containerName="istio-proxy"
Oct 31 13:52:56.799378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.799263    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6349679a-67f7-41e5-b289-dc91302b5977" containerName="istio-proxy"
Oct 31 13:52:56.799378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.799297    4428 cpu_manager.go:414] "containerMap: removing container" podUID="1821cf0c-d453-4c5c-83bd-7c3e22666953" containerName="payload-pre-processing"
Oct 31 13:52:56.799378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.799303    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1821cf0c-d453-4c5c-83bd-7c3e22666953" containerName="payload-pre-processing"
Oct 31 13:52:56.799522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.799430    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1821cf0c-d453-4c5c-83bd-7c3e22666953" containerName="payload-pre-processing"
Oct 31 13:52:56.799522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.799448    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6349679a-67f7-41e5-b289-dc91302b5977" containerName="istio-proxy"
Oct 31 13:52:56.799522 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.799462    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="803c3db4-5311-4a96-8474-64304a5995ba" containerName="payload-processing"
Oct 31 13:52:56.803595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.803575    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:52:56.820449 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.816057    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-20b8a4-serving-cert\""
Oct 31 13:52:56.821397 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.821365    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"]
Oct 31 13:52:56.907239 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.907199    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:56.920278 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.920237    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sk6t\" (UniqueName: \"kubernetes.io/projected/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-kube-api-access-9sk6t\") pod \"maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626\" (UID: \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:52:56.920483 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.920299    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-metrics-certs\") pod \"maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626\" (UID: \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:52:56.920536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.920494    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626\" (UID: \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:52:56.920576 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:56.920561    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626\" (UID: \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:52:57.031897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:57.024063    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sk6t\" (UniqueName: \"kubernetes.io/projected/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-kube-api-access-9sk6t\") pod \"maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626\" (UID: \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:52:57.031897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:57.024137    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-metrics-certs\") pod \"maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626\" (UID: \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:52:57.031897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:57.024277    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626\" (UID: \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:52:57.031897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:57.024342    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626\" (UID: \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:52:57.047009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:57.042589    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-metrics-certs\") pod \"maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626\" (UID: \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:52:57.051933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:57.049776    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626\" (UID: \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:52:57.051933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:57.051192    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626\" (UID: \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:52:57.065559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:57.065519    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:57.070490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:57.070401    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sk6t\" (UniqueName: \"kubernetes.io/projected/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-kube-api-access-9sk6t\") pod \"maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626\" (UID: \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:52:57.134248 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:57.134214    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:52:57.426162 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:52:57.426131    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39928dfc_c155_42d2_8ab0_8a21e0bcb0d4.slice/crio-154c1dcda7026099861136fae8e70eddcb8c182f3c085e08154491c5bcd41625 WatchSource:0}: Error finding container 154c1dcda7026099861136fae8e70eddcb8c182f3c085e08154491c5bcd41625: Status 404 returned error can't find the container with id 154c1dcda7026099861136fae8e70eddcb8c182f3c085e08154491c5bcd41625
Oct 31 13:52:57.427475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:57.427447    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"]
Oct 31 13:52:57.427668 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:57.427652    4428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:52:57.855119 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:57.855085    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11125-b2d66a"]
Oct 31 13:52:57.861306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:57.861277    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11125-b2d66a"
Oct 31 13:52:57.867755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:57.867718    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11125-b2d66a"]
Oct 31 13:52:57.869657 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:57.869624    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-whw9x\""
Oct 31 13:52:57.946891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:57.946819    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:58.004444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:58.004411    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp5kd\" (UniqueName: \"kubernetes.io/projected/9ec48f74-8d2b-4680-8e00-1ccd2ac03c32-kube-api-access-bp5kd\") pod \"test-curl-11125-b2d66a\" (UID: \"9ec48f74-8d2b-4680-8e00-1ccd2ac03c32\") " pod="opendatahub/test-curl-11125-b2d66a"
Oct 31 13:52:58.105984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:58.105913    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp5kd\" (UniqueName: \"kubernetes.io/projected/9ec48f74-8d2b-4680-8e00-1ccd2ac03c32-kube-api-access-bp5kd\") pod \"test-curl-11125-b2d66a\" (UID: \"9ec48f74-8d2b-4680-8e00-1ccd2ac03c32\") " pod="opendatahub/test-curl-11125-b2d66a"
Oct 31 13:52:58.120144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:58.120119    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp5kd\" (UniqueName: \"kubernetes.io/projected/9ec48f74-8d2b-4680-8e00-1ccd2ac03c32-kube-api-access-bp5kd\") pod \"test-curl-11125-b2d66a\" (UID: \"9ec48f74-8d2b-4680-8e00-1ccd2ac03c32\") " pod="opendatahub/test-curl-11125-b2d66a"
Oct 31 13:52:58.181011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:58.180956    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11125-b2d66a"
Oct 31 13:52:58.406527 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:58.406432    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626" event={"ID":"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4","Type":"ContainerStarted","Data":"8828d2ddd74e3321c86a14b4584a1de5e073af5f691bc19053af81b902be13e2"}
Oct 31 13:52:58.406527 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:58.406492    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626" event={"ID":"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4","Type":"ContainerStarted","Data":"154c1dcda7026099861136fae8e70eddcb8c182f3c085e08154491c5bcd41625"}
Oct 31 13:52:58.406527 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:58.406520    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:52:58.435819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:58.435765    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626" podStartSLOduration=1.853877705 podStartE2EDuration="2.435749576s" totalImagesPullingTime="581.871871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:52:58.431404604 +0000 UTC m=+2975.829753230" watchObservedRunningTime="2026-10-31 13:52:58.435749576 +0000 UTC m=+2975.834098201"
Oct 31 13:52:58.470359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:58.470327    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11125-b2d66a"]
Oct 31 13:52:58.470800 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:52:58.470757    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ec48f74_8d2b_4680_8e00_1ccd2ac03c32.slice/crio-d3b32315adb40c7204dbc7582f9cf1f70ea777fd3b6ca76fec7ab79488917134 WatchSource:0}: Error finding container d3b32315adb40c7204dbc7582f9cf1f70ea777fd3b6ca76fec7ab79488917134: Status 404 returned error can't find the container with id d3b32315adb40c7204dbc7582f9cf1f70ea777fd3b6ca76fec7ab79488917134
Oct 31 13:52:58.829063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:58.829017    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:52:59.415250 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:59.415208    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11125-b2d66a" event={"ID":"9ec48f74-8d2b-4680-8e00-1ccd2ac03c32","Type":"ContainerStarted","Data":"d3b32315adb40c7204dbc7582f9cf1f70ea777fd3b6ca76fec7ab79488917134"}
Oct 31 13:52:59.591309 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:52:59.591210    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:00.336898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:00.329135    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:00.424970 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:00.424922    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11125-b2d66a" event={"ID":"9ec48f74-8d2b-4680-8e00-1ccd2ac03c32","Type":"ContainerStarted","Data":"9bc71c3a0233e30ae5ae609fa3b1265f37e2101c9d20625b5ca04be7fbaa4c0e"}
Oct 31 13:53:00.449367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:00.449299    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11125-b2d66a" podStartSLOduration=2.325282945 podStartE2EDuration="3.449284033s" totalImagesPullingTime="1.124001088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:00.446660392 +0000 UTC m=+2977.845009017" watchObservedRunningTime="2026-10-31 13:53:00.449284033 +0000 UTC m=+2977.847632657"
Oct 31 13:53:01.163664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:01.163623    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11125-b2d66a"]
Oct 31 13:53:01.169405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:01.169375    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11125-b2d66a"]
Oct 31 13:53:01.276032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:01.275996    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:02.102893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:02.102636    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:02.441096 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:02.441006    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11125-b2d66a" podUID="9ec48f74-8d2b-4680-8e00-1ccd2ac03c32" containerName="test-curl-11125-b2d66a" containerID="cri-o://9bc71c3a0233e30ae5ae609fa3b1265f37e2101c9d20625b5ca04be7fbaa4c0e" gracePeriod=2
Oct 31 13:53:02.443510 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:53:02.443483    4428 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11125-b2d66a\" is forbidden: User \"system:node:ip-10-0-136-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-168.us-east-2.compute.internal' and this object" podUID="9ec48f74-8d2b-4680-8e00-1ccd2ac03c32" pod="opendatahub/test-curl-11125-b2d66a"
Oct 31 13:53:02.754472 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:53:02.754437    4428 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11125-b2d66a\" is forbidden: User \"system:node:ip-10-0-136-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-168.us-east-2.compute.internal' and this object" podUID="9ec48f74-8d2b-4680-8e00-1ccd2ac03c32" pod="opendatahub/test-curl-11125-b2d66a"
Oct 31 13:53:02.897895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:02.896587    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:03.668894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:03.664760    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:04.421282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:04.421250    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:53:04.495567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:04.495470    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:04.766526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:04.766498    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11125-b2d66a"
Oct 31 13:53:04.903711 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:04.903676    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec48f74-8d2b-4680-8e00-1ccd2ac03c32-kube-api-access-bp5kd\" (UniqueName: \"kubernetes.io/projected/9ec48f74-8d2b-4680-8e00-1ccd2ac03c32-kube-api-access-bp5kd\") pod \"9ec48f74-8d2b-4680-8e00-1ccd2ac03c32\" (UID: \"9ec48f74-8d2b-4680-8e00-1ccd2ac03c32\") "
Oct 31 13:53:04.906570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:04.906531    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec48f74-8d2b-4680-8e00-1ccd2ac03c32-kube-api-access-bp5kd" pod "9ec48f74-8d2b-4680-8e00-1ccd2ac03c32" (UID: "9ec48f74-8d2b-4680-8e00-1ccd2ac03c32"). InnerVolumeSpecName "kube-api-access-bp5kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:53:05.006721 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:05.006628    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp5kd\" (UniqueName: \"kubernetes.io/projected/9ec48f74-8d2b-4680-8e00-1ccd2ac03c32-kube-api-access-bp5kd\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:05.295136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:05.295049    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:05.463581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:05.463547    4428 generic.go:356] "Generic (PLEG): container finished" podID="9ec48f74-8d2b-4680-8e00-1ccd2ac03c32" containerID="9bc71c3a0233e30ae5ae609fa3b1265f37e2101c9d20625b5ca04be7fbaa4c0e" exitCode=137
Oct 31 13:53:05.463767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:05.463597    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11125-b2d66a"
Oct 31 13:53:05.463767 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:05.463651    4428 scope.go:122] "RemoveContainer" containerID="9bc71c3a0233e30ae5ae609fa3b1265f37e2101c9d20625b5ca04be7fbaa4c0e"
Oct 31 13:53:05.476194 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:53:05.476028    4428 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11125-b2d66a\" is forbidden: User \"system:node:ip-10-0-136-168.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-168.us-east-2.compute.internal' and this object" podUID="9ec48f74-8d2b-4680-8e00-1ccd2ac03c32" pod="opendatahub/test-curl-11125-b2d66a"
Oct 31 13:53:05.476265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:05.476216    4428 scope.go:122] "RemoveContainer" containerID="9bc71c3a0233e30ae5ae609fa3b1265f37e2101c9d20625b5ca04be7fbaa4c0e"
Oct 31 13:53:05.476521 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:53:05.476502    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bc71c3a0233e30ae5ae609fa3b1265f37e2101c9d20625b5ca04be7fbaa4c0e\": container with ID starting with 9bc71c3a0233e30ae5ae609fa3b1265f37e2101c9d20625b5ca04be7fbaa4c0e not found: ID does not exist" containerID="9bc71c3a0233e30ae5ae609fa3b1265f37e2101c9d20625b5ca04be7fbaa4c0e"
Oct 31 13:53:05.476583 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:05.476529    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc71c3a0233e30ae5ae609fa3b1265f37e2101c9d20625b5ca04be7fbaa4c0e"} err="failed to get container status \"9bc71c3a0233e30ae5ae609fa3b1265f37e2101c9d20625b5ca04be7fbaa4c0e\": rpc error: code = NotFound desc = could not find container \"9bc71c3a0233e30ae5ae609fa3b1265f37e2101c9d20625b5ca04be7fbaa4c0e\": container with ID starting with 9bc71c3a0233e30ae5ae609fa3b1265f37e2101c9d20625b5ca04be7fbaa4c0e not found: ID does not exist"
Oct 31 13:53:06.195662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:06.195606    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:06.758076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:06.758044    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec48f74-8d2b-4680-8e00-1ccd2ac03c32" path="/var/lib/kubelet/pods/9ec48f74-8d2b-4680-8e00-1ccd2ac03c32/volumes"
Oct 31 13:53:07.028133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:07.028023    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:08.038954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:08.031499    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:08.791560 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:08.791519    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:09.634068 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:09.634024    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:10.579159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:10.579124    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:11.505893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:11.494622    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:12.419903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:12.419736    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:13.265191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:13.265152    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:14.203347 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:14.203312    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:14.829177 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:14.829141    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:15.042430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:15.042397    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:15.644566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:15.644523    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:15.929776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:15.929672    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:16.728625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:16.728585    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:16.785348 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:16.785315    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"]
Oct 31 13:53:16.785648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:16.785523    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj" podUID="208fddf2-00ef-4f4a-944a-f3d93ba37911" containerName="istio-proxy" containerID="cri-o://bf98b31a9eb58fa0044efe091109b4b209696764cdd0458e39c2166557da6900" gracePeriod=30
Oct 31 13:53:16.813445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:16.813407    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"]
Oct 31 13:53:16.813710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:16.813673    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy" containerID="cri-o://de4ab4fad18c7d488c8dc5b9464b2c326ab943da71609bd6be99190bdeca3980" gracePeriod=30
Oct 31 13:53:17.213557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:17.213513    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"]
Oct 31 13:53:17.214151 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:17.213747    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld" podUID="6002eb05-9237-48f2-972a-448248703ca4" containerName="istio-proxy" containerID="cri-o://f6d17cdd09c7023eefe9f5e0d29eba4024fa3123d0ad91241fea7b3575f4dd03" gracePeriod=30
Oct 31 13:53:17.232411 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:17.232323    4428 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:53:17.232629 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:17.232410    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:53:17.896449 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:17.896119    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:18.029081 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:18.029041    4428 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:53:18.029281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:18.029112    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld" podUID="6002eb05-9237-48f2-972a-448248703ca4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:53:18.742994 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:53:18.742954    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:53:19.055393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:19.055286    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:19.227413 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:19.227376    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:19.418455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:19.418336    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:19.557928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:19.554418    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"]
Oct 31 13:53:19.597891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:19.597454    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:19.971840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:19.970663    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:20.300653 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:20.300561    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:20.629141 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:20.629053    4428 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:53:20.629445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:20.629403    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj" podUID="208fddf2-00ef-4f4a-944a-f3d93ba37911" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:53:21.247891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:21.246664    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:21.996014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:21.995975    4428 generic.go:356] "Generic (PLEG): container finished" podID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerID="de4ab4fad18c7d488c8dc5b9464b2c326ab943da71609bd6be99190bdeca3980" exitCode=0
Oct 31 13:53:21.996014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:21.996007    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" event={"ID":"1667cc7b-35eb-4c92-9fa7-239aaa5990d1","Type":"ContainerDied","Data":"de4ab4fad18c7d488c8dc5b9464b2c326ab943da71609bd6be99190bdeca3980"}
Oct 31 13:53:21.996301 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:21.996082    4428 scope.go:122] "RemoveContainer" containerID="71bc7b5aabe8ff9a363889f5436413ba9693154a01dcd88ae0d9437b88bf20f4"
Oct 31 13:53:21.998524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:21.998496    4428 generic.go:356] "Generic (PLEG): container finished" podID="208fddf2-00ef-4f4a-944a-f3d93ba37911" containerID="bf98b31a9eb58fa0044efe091109b4b209696764cdd0458e39c2166557da6900" exitCode=0
Oct 31 13:53:21.998614 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:21.998558    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj" event={"ID":"208fddf2-00ef-4f4a-944a-f3d93ba37911","Type":"ContainerDied","Data":"bf98b31a9eb58fa0044efe091109b4b209696764cdd0458e39c2166557da6900"}
Oct 31 13:53:22.148343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.148308    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:22.158596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.158549    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:53:22.320024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.319991    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-credential-socket\") pod \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") "
Oct 31 13:53:22.320502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.320104    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-envoy\") pod \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") "
Oct 31 13:53:22.320502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.320167    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-token\" (UniqueName: \"kubernetes.io/projected/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-token\") pod \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") "
Oct 31 13:53:22.320502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.320344    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-podinfo\") pod \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") "
Oct 31 13:53:22.320502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.320397    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istiod-ca-cert\") pod \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") "
Oct 31 13:53:22.320502 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.320410    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-credential-socket" pod "1667cc7b-35eb-4c92-9fa7-239aaa5990d1" (UID: "1667cc7b-35eb-4c92-9fa7-239aaa5990d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.320905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.320541    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-kube-api-access-rw8nc\" (UniqueName: \"kubernetes.io/projected/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-kube-api-access-rw8nc\") pod \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") "
Oct 31 13:53:22.320905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.320595    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-workload-certs\") pod \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") "
Oct 31 13:53:22.320905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.320666    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-workload-socket\") pod \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") "
Oct 31 13:53:22.320905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.320717    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-data\") pod \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\" (UID: \"1667cc7b-35eb-4c92-9fa7-239aaa5990d1\") "
Oct 31 13:53:22.321360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.321292    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-workload-certs" pod "1667cc7b-35eb-4c92-9fa7-239aaa5990d1" (UID: "1667cc7b-35eb-4c92-9fa7-239aaa5990d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.321538 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.321513    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istiod-ca-cert" pod "1667cc7b-35eb-4c92-9fa7-239aaa5990d1" (UID: "1667cc7b-35eb-4c92-9fa7-239aaa5990d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:53:22.321746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.321722    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-workload-socket" pod "1667cc7b-35eb-4c92-9fa7-239aaa5990d1" (UID: "1667cc7b-35eb-4c92-9fa7-239aaa5990d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.321819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.321753    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-data" pod "1667cc7b-35eb-4c92-9fa7-239aaa5990d1" (UID: "1667cc7b-35eb-4c92-9fa7-239aaa5990d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.322023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.321999    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.322106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.322026    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.322106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.322041    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.322106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.322057    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.322106 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.322072    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.323399 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.323376    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-token" pod "1667cc7b-35eb-4c92-9fa7-239aaa5990d1" (UID: "1667cc7b-35eb-4c92-9fa7-239aaa5990d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:53:22.323788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.323736    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-podinfo" pod "1667cc7b-35eb-4c92-9fa7-239aaa5990d1" (UID: "1667cc7b-35eb-4c92-9fa7-239aaa5990d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:53:22.324235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.324193    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-envoy" pod "1667cc7b-35eb-4c92-9fa7-239aaa5990d1" (UID: "1667cc7b-35eb-4c92-9fa7-239aaa5990d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.324235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.324198    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-kube-api-access-rw8nc" pod "1667cc7b-35eb-4c92-9fa7-239aaa5990d1" (UID: "1667cc7b-35eb-4c92-9fa7-239aaa5990d1"). InnerVolumeSpecName "kube-api-access-rw8nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:53:22.424078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.424037    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw8nc\" (UniqueName: \"kubernetes.io/projected/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-kube-api-access-rw8nc\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.424229 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.424084    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.424229 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.424104    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.424229 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.424118    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1667cc7b-35eb-4c92-9fa7-239aaa5990d1-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.553547 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.553519    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:53:22.578543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.578516    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.626646    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6002eb05-9237-48f2-972a-448248703ca4-istio-token\" (UniqueName: \"kubernetes.io/projected/6002eb05-9237-48f2-972a-448248703ca4-istio-token\") pod \"6002eb05-9237-48f2-972a-448248703ca4\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") "
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.626713    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-istio-envoy\") pod \"6002eb05-9237-48f2-972a-448248703ca4\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") "
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.626762    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/208fddf2-00ef-4f4a-944a-f3d93ba37911-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208fddf2-00ef-4f4a-944a-f3d93ba37911-istiod-ca-cert\") pod \"208fddf2-00ef-4f4a-944a-f3d93ba37911\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") "
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.626803    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-workload-socket\") pod \"208fddf2-00ef-4f4a-944a-f3d93ba37911\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") "
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.626922    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-token\" (UniqueName: \"kubernetes.io/projected/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-token\") pod \"208fddf2-00ef-4f4a-944a-f3d93ba37911\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") "
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.626997    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-workload-certs\") pod \"208fddf2-00ef-4f4a-944a-f3d93ba37911\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") "
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.627029    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-istio-data\") pod \"6002eb05-9237-48f2-972a-448248703ca4\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") "
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.627110    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-envoy\") pod \"208fddf2-00ef-4f4a-944a-f3d93ba37911\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") "
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.627143    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-credential-socket\") pod \"6002eb05-9237-48f2-972a-448248703ca4\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") "
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.627238    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6002eb05-9237-48f2-972a-448248703ca4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6002eb05-9237-48f2-972a-448248703ca4-istio-podinfo\") pod \"6002eb05-9237-48f2-972a-448248703ca4\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") "
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.627290    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6002eb05-9237-48f2-972a-448248703ca4-kube-api-access-qcsjt\" (UniqueName: \"kubernetes.io/projected/6002eb05-9237-48f2-972a-448248703ca4-kube-api-access-qcsjt\") pod \"6002eb05-9237-48f2-972a-448248703ca4\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") "
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.627339    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208fddf2-00ef-4f4a-944a-f3d93ba37911-kube-api-access-fb89t\" (UniqueName: \"kubernetes.io/projected/208fddf2-00ef-4f4a-944a-f3d93ba37911-kube-api-access-fb89t\") pod \"208fddf2-00ef-4f4a-944a-f3d93ba37911\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") "
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.627381    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-workload-certs\") pod \"6002eb05-9237-48f2-972a-448248703ca4\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") "
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.627453    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-podinfo\") pod \"208fddf2-00ef-4f4a-944a-f3d93ba37911\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") "
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.627473    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-workload-socket\") pod \"6002eb05-9237-48f2-972a-448248703ca4\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") "
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.627504    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6002eb05-9237-48f2-972a-448248703ca4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6002eb05-9237-48f2-972a-448248703ca4-istiod-ca-cert\") pod \"6002eb05-9237-48f2-972a-448248703ca4\" (UID: \"6002eb05-9237-48f2-972a-448248703ca4\") "
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.627530    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-credential-socket\") pod \"208fddf2-00ef-4f4a-944a-f3d93ba37911\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") "
Oct 31 13:53:22.627802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.627567    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-data\") pod \"208fddf2-00ef-4f4a-944a-f3d93ba37911\" (UID: \"208fddf2-00ef-4f4a-944a-f3d93ba37911\") "
Oct 31 13:53:22.629289 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.629253    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-data" pod "208fddf2-00ef-4f4a-944a-f3d93ba37911" (UID: "208fddf2-00ef-4f4a-944a-f3d93ba37911"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.629485 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.629455    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-workload-certs" pod "6002eb05-9237-48f2-972a-448248703ca4" (UID: "6002eb05-9237-48f2-972a-448248703ca4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.629794 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.629560    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-workload-socket" pod "6002eb05-9237-48f2-972a-448248703ca4" (UID: "6002eb05-9237-48f2-972a-448248703ca4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.629921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.629825    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6002eb05-9237-48f2-972a-448248703ca4-istiod-ca-cert" pod "6002eb05-9237-48f2-972a-448248703ca4" (UID: "6002eb05-9237-48f2-972a-448248703ca4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:53:22.630229 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.630145    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-workload-certs" pod "208fddf2-00ef-4f4a-944a-f3d93ba37911" (UID: "208fddf2-00ef-4f4a-944a-f3d93ba37911"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.630324 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.630239    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-credential-socket" pod "208fddf2-00ef-4f4a-944a-f3d93ba37911" (UID: "208fddf2-00ef-4f4a-944a-f3d93ba37911"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.630479 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.630443    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-workload-socket" pod "208fddf2-00ef-4f4a-944a-f3d93ba37911" (UID: "208fddf2-00ef-4f4a-944a-f3d93ba37911"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.630807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.630782    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208fddf2-00ef-4f4a-944a-f3d93ba37911-istiod-ca-cert" pod "208fddf2-00ef-4f4a-944a-f3d93ba37911" (UID: "208fddf2-00ef-4f4a-944a-f3d93ba37911"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:53:22.631569 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.631488    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-istio-envoy" pod "6002eb05-9237-48f2-972a-448248703ca4" (UID: "6002eb05-9237-48f2-972a-448248703ca4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.631818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.631793    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-credential-socket" pod "6002eb05-9237-48f2-972a-448248703ca4" (UID: "6002eb05-9237-48f2-972a-448248703ca4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.632100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.632076    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-istio-data" pod "6002eb05-9237-48f2-972a-448248703ca4" (UID: "6002eb05-9237-48f2-972a-448248703ca4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.633391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.633352    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-envoy" pod "208fddf2-00ef-4f4a-944a-f3d93ba37911" (UID: "208fddf2-00ef-4f4a-944a-f3d93ba37911"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:22.635442 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.635417    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-podinfo" pod "208fddf2-00ef-4f4a-944a-f3d93ba37911" (UID: "208fddf2-00ef-4f4a-944a-f3d93ba37911"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:53:22.636769 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.636745    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-token" pod "208fddf2-00ef-4f4a-944a-f3d93ba37911" (UID: "208fddf2-00ef-4f4a-944a-f3d93ba37911"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:53:22.637147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.637104    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6002eb05-9237-48f2-972a-448248703ca4-kube-api-access-qcsjt" pod "6002eb05-9237-48f2-972a-448248703ca4" (UID: "6002eb05-9237-48f2-972a-448248703ca4"). InnerVolumeSpecName "kube-api-access-qcsjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:53:22.637336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.637314    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6002eb05-9237-48f2-972a-448248703ca4-istio-podinfo" pod "6002eb05-9237-48f2-972a-448248703ca4" (UID: "6002eb05-9237-48f2-972a-448248703ca4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:53:22.638816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.638766    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6002eb05-9237-48f2-972a-448248703ca4-istio-token" pod "6002eb05-9237-48f2-972a-448248703ca4" (UID: "6002eb05-9237-48f2-972a-448248703ca4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:53:22.640393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.640362    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208fddf2-00ef-4f4a-944a-f3d93ba37911-kube-api-access-fb89t" pod "208fddf2-00ef-4f4a-944a-f3d93ba37911" (UID: "208fddf2-00ef-4f4a-944a-f3d93ba37911"). InnerVolumeSpecName "kube-api-access-fb89t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:53:22.729682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729641    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.729682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729679    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.729682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729691    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6002eb05-9237-48f2-972a-448248703ca4-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.729956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729701    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.729956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729712    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.729956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729725    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6002eb05-9237-48f2-972a-448248703ca4-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.729956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729738    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.729956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729749    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208fddf2-00ef-4f4a-944a-f3d93ba37911-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.729956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729758    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.729956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729766    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.729956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729776    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.729956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729784    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.729956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729792    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208fddf2-00ef-4f4a-944a-f3d93ba37911-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.729956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729800    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.729956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729808    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6002eb05-9237-48f2-972a-448248703ca4-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.729956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729818    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcsjt\" (UniqueName: \"kubernetes.io/projected/6002eb05-9237-48f2-972a-448248703ca4-kube-api-access-qcsjt\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.729956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729827    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb89t\" (UniqueName: \"kubernetes.io/projected/208fddf2-00ef-4f4a-944a-f3d93ba37911-kube-api-access-fb89t\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:22.729956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:22.729835    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6002eb05-9237-48f2-972a-448248703ca4-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:23.007460 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.007428    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"
Oct 31 13:53:23.007668 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.007462    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj" event={"ID":"208fddf2-00ef-4f4a-944a-f3d93ba37911","Type":"ContainerDied","Data":"68011f1f86b5e83e9ba011efb69bd13c889b66596544232fe3e37f10baa0af0d"}
Oct 31 13:53:23.007668 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.007524    4428 scope.go:122] "RemoveContainer" containerID="bf98b31a9eb58fa0044efe091109b4b209696764cdd0458e39c2166557da6900"
Oct 31 13:53:23.009925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.009212    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"
Oct 31 13:53:23.009925 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.009409    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns" event={"ID":"1667cc7b-35eb-4c92-9fa7-239aaa5990d1","Type":"ContainerDied","Data":"bd11ad3de22abdaa84ae8db21fcd82ff0c72f34cc52bb696260ad04dccfaaa77"}
Oct 31 13:53:23.013807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.013780    4428 generic.go:356] "Generic (PLEG): container finished" podID="6002eb05-9237-48f2-972a-448248703ca4" containerID="f6d17cdd09c7023eefe9f5e0d29eba4024fa3123d0ad91241fea7b3575f4dd03" exitCode=0
Oct 31 13:53:23.013983 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.013847    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"
Oct 31 13:53:23.014064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.013976    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld" event={"ID":"6002eb05-9237-48f2-972a-448248703ca4","Type":"ContainerDied","Data":"f6d17cdd09c7023eefe9f5e0d29eba4024fa3123d0ad91241fea7b3575f4dd03"}
Oct 31 13:53:23.014064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.014022    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld" event={"ID":"6002eb05-9237-48f2-972a-448248703ca4","Type":"ContainerDied","Data":"052cf1c2b9fd3545859b3549b83b49884128bbba2cc269822c7e1b99616cfe9f"}
Oct 31 13:53:23.022271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.022249    4428 scope.go:122] "RemoveContainer" containerID="de4ab4fad18c7d488c8dc5b9464b2c326ab943da71609bd6be99190bdeca3980"
Oct 31 13:53:23.033411 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.033392    4428 scope.go:122] "RemoveContainer" containerID="f6d17cdd09c7023eefe9f5e0d29eba4024fa3123d0ad91241fea7b3575f4dd03"
Oct 31 13:53:23.044679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.044651    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"]
Oct 31 13:53:23.051138 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.051110    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-9hgns"]
Oct 31 13:53:23.056279 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.056261    4428 scope.go:122] "RemoveContainer" containerID="86f4b0c34be4808cb41009e58c6fb3a9ddd7aab94ae09bd4c85399f1e7fa1580"
Oct 31 13:53:23.066538 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.066513    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"]
Oct 31 13:53:23.068277 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.068251    4428 scope.go:122] "RemoveContainer" containerID="f6d17cdd09c7023eefe9f5e0d29eba4024fa3123d0ad91241fea7b3575f4dd03"
Oct 31 13:53:23.068630 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:53:23.068605    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d17cdd09c7023eefe9f5e0d29eba4024fa3123d0ad91241fea7b3575f4dd03\": container with ID starting with f6d17cdd09c7023eefe9f5e0d29eba4024fa3123d0ad91241fea7b3575f4dd03 not found: ID does not exist" containerID="f6d17cdd09c7023eefe9f5e0d29eba4024fa3123d0ad91241fea7b3575f4dd03"
Oct 31 13:53:23.068723 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.068639    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d17cdd09c7023eefe9f5e0d29eba4024fa3123d0ad91241fea7b3575f4dd03"} err="failed to get container status \"f6d17cdd09c7023eefe9f5e0d29eba4024fa3123d0ad91241fea7b3575f4dd03\": rpc error: code = NotFound desc = could not find container \"f6d17cdd09c7023eefe9f5e0d29eba4024fa3123d0ad91241fea7b3575f4dd03\": container with ID starting with f6d17cdd09c7023eefe9f5e0d29eba4024fa3123d0ad91241fea7b3575f4dd03 not found: ID does not exist"
Oct 31 13:53:23.068723 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.068668    4428 scope.go:122] "RemoveContainer" containerID="86f4b0c34be4808cb41009e58c6fb3a9ddd7aab94ae09bd4c85399f1e7fa1580"
Oct 31 13:53:23.069110 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:53:23.069077    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f4b0c34be4808cb41009e58c6fb3a9ddd7aab94ae09bd4c85399f1e7fa1580\": container with ID starting with 86f4b0c34be4808cb41009e58c6fb3a9ddd7aab94ae09bd4c85399f1e7fa1580 not found: ID does not exist" containerID="86f4b0c34be4808cb41009e58c6fb3a9ddd7aab94ae09bd4c85399f1e7fa1580"
Oct 31 13:53:23.069209 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.069117    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f4b0c34be4808cb41009e58c6fb3a9ddd7aab94ae09bd4c85399f1e7fa1580"} err="failed to get container status \"86f4b0c34be4808cb41009e58c6fb3a9ddd7aab94ae09bd4c85399f1e7fa1580\": rpc error: code = NotFound desc = could not find container \"86f4b0c34be4808cb41009e58c6fb3a9ddd7aab94ae09bd4c85399f1e7fa1580\": container with ID starting with 86f4b0c34be4808cb41009e58c6fb3a9ddd7aab94ae09bd4c85399f1e7fa1580 not found: ID does not exist"
Oct 31 13:53:23.078593 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.078548    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-4bmkj"]
Oct 31 13:53:23.102903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.101390    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:23.129805 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.127794    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"]
Oct 31 13:53:23.139832 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.136223    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-6wcld"]
Oct 31 13:53:23.898595 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:23.898544    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:24.768515 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:24.768460    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" path="/var/lib/kubelet/pods/1667cc7b-35eb-4c92-9fa7-239aaa5990d1/volumes"
Oct 31 13:53:24.769190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:24.769166    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="208fddf2-00ef-4f4a-944a-f3d93ba37911" path="/var/lib/kubelet/pods/208fddf2-00ef-4f4a-944a-f3d93ba37911/volumes"
Oct 31 13:53:24.769771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:24.769749    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6002eb05-9237-48f2-972a-448248703ca4" path="/var/lib/kubelet/pods/6002eb05-9237-48f2-972a-448248703ca4/volumes"
Oct 31 13:53:24.822993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:24.821416    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:25.162771 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:25.162486    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:25.876133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:25.875283    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:26.034716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:26.034684    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:26.394861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:26.394830    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:26.581803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:26.581766    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:27.014950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:27.014371    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:27.410796 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:27.410197    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"]
Oct 31 13:53:27.533896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:27.529803    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:28.237891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:28.230763    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:29.140530 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:29.138208    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:29.441854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:29.441744    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"]
Oct 31 13:53:29.476586 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:29.476546    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:29.665890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:29.665453    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:30.131157 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:30.131124    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:30.326368 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:30.326327    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:30.580533 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:30.580484    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:30.746938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:30.745991    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:30.909578 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:30.909467    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:31.097951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:31.096198    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:31.671892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:31.671335    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:32.654911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:32.648680    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:33.875681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:33.875642    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:34.598624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:34.598592    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:34.855517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:34.847179    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:34.926125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:34.926076    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4" podUID="38ef9813-0afd-477c-8e9f-1bcbfbfd17a4" containerName="main" containerID="cri-o://f430bea93637a6d60cb8fef4b705aa7fe0531b7b43e35d8ceb5f07e22258b808" gracePeriod=45
Oct 31 13:53:35.207794 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.207750    4428 generic.go:356] "Generic (PLEG): container finished" podID="38ef9813-0afd-477c-8e9f-1bcbfbfd17a4" containerID="f430bea93637a6d60cb8fef4b705aa7fe0531b7b43e35d8ceb5f07e22258b808" exitCode=0
Oct 31 13:53:35.208000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.207816    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4" event={"ID":"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4","Type":"ContainerDied","Data":"f430bea93637a6d60cb8fef4b705aa7fe0531b7b43e35d8ceb5f07e22258b808"}
Oct 31 13:53:35.257499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.257472    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:53:35.351248 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.351214    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-tmp-dir\") pod \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") "
Oct 31 13:53:35.351466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.351279    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-home\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-home\") pod \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") "
Oct 31 13:53:35.351466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.351313    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-tls-certs\") pod \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") "
Oct 31 13:53:35.351466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.351369    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-dshm\") pod \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") "
Oct 31 13:53:35.351466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.351414    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-model-cache\") pod \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") "
Oct 31 13:53:35.351696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.351567    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-tmp-dir" pod "38ef9813-0afd-477c-8e9f-1bcbfbfd17a4" (UID: "38ef9813-0afd-477c-8e9f-1bcbfbfd17a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:35.351696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.351581    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-kube-api-access-d4mz9\" (UniqueName: \"kubernetes.io/projected/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-kube-api-access-d4mz9\") pod \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\" (UID: \"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4\") "
Oct 31 13:53:35.351696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.351607    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-home" pod "38ef9813-0afd-477c-8e9f-1bcbfbfd17a4" (UID: "38ef9813-0afd-477c-8e9f-1bcbfbfd17a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:35.351850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.351731    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-model-cache" pod "38ef9813-0afd-477c-8e9f-1bcbfbfd17a4" (UID: "38ef9813-0afd-477c-8e9f-1bcbfbfd17a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:35.352430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.352406    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:35.352534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.352433    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:35.352534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.352446    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:35.353909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.353858    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-dshm" pod "38ef9813-0afd-477c-8e9f-1bcbfbfd17a4" (UID: "38ef9813-0afd-477c-8e9f-1bcbfbfd17a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:35.354014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.353909    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-tls-certs" pod "38ef9813-0afd-477c-8e9f-1bcbfbfd17a4" (UID: "38ef9813-0afd-477c-8e9f-1bcbfbfd17a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:53:35.354014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.353975    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-kube-api-access-d4mz9" pod "38ef9813-0afd-477c-8e9f-1bcbfbfd17a4" (UID: "38ef9813-0afd-477c-8e9f-1bcbfbfd17a4"). InnerVolumeSpecName "kube-api-access-d4mz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:53:35.454289 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.454200    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4mz9\" (UniqueName: \"kubernetes.io/projected/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-kube-api-access-d4mz9\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:35.454289 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.454229    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:35.454289 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.454241    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:35.701788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.701747    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:35.923127 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:35.923096    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:36.091894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:36.089980    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:36.281895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:36.281008    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4" event={"ID":"38ef9813-0afd-477c-8e9f-1bcbfbfd17a4","Type":"ContainerDied","Data":"9ea6937c19cd76e735357bc272b018577d344593a13c75195c50195cceabaee6"}
Oct 31 13:53:36.281895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:36.281075    4428 scope.go:122] "RemoveContainer" containerID="f430bea93637a6d60cb8fef4b705aa7fe0531b7b43e35d8ceb5f07e22258b808"
Oct 31 13:53:36.281895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:36.281265    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"
Oct 31 13:53:36.324007 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:36.323976    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"]
Oct 31 13:53:36.328795 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:36.328768    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6a3533/auth-test-model-6a3533-kserve-679c56786f-rp8b4"]
Oct 31 13:53:36.719898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:36.716855    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:36.758159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:36.758127    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ef9813-0afd-477c-8e9f-1bcbfbfd17a4" path="/var/lib/kubelet/pods/38ef9813-0afd-477c-8e9f-1bcbfbfd17a4/volumes"
Oct 31 13:53:37.696948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:37.695173    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:38.522893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:38.521602    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:39.433957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:39.433920    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:40.263375 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:40.263329    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:40.880181 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:40.880142    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:41.048020 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:41.047967    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:41.625410 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:41.623163    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:42.427942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.424543    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:42.450288 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.449651    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q" podUID="2ee33e40-dba2-418c-97f0-f120d64a81f9" containerName="main" containerID="cri-o://0345a1a15b79c25cc4f30462ccf6cb6c9cc577cf362327dfc387a392c0a38f6d" gracePeriod=45
Oct 31 13:53:42.577394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.577350    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q" podUID="2ee33e40-dba2-418c-97f0-f120d64a81f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.228:8000/ready\": dial tcp 10.134.0.228:8000: connect: connection refused"
Oct 31 13:53:42.753921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.753886    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:53:42.841409 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.841375    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ee33e40-dba2-418c-97f0-f120d64a81f9-kube-api-access-8vmrf\" (UniqueName: \"kubernetes.io/projected/2ee33e40-dba2-418c-97f0-f120d64a81f9-kube-api-access-8vmrf\") pod \"2ee33e40-dba2-418c-97f0-f120d64a81f9\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") "
Oct 31 13:53:42.841599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.841548    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-tmp-dir\") pod \"2ee33e40-dba2-418c-97f0-f120d64a81f9\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") "
Oct 31 13:53:42.841599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.841590    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-home\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-home\") pod \"2ee33e40-dba2-418c-97f0-f120d64a81f9\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") "
Oct 31 13:53:42.841677 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.841634    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ee33e40-dba2-418c-97f0-f120d64a81f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ee33e40-dba2-418c-97f0-f120d64a81f9-tls-certs\") pod \"2ee33e40-dba2-418c-97f0-f120d64a81f9\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") "
Oct 31 13:53:42.841714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.841675    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-dshm\") pod \"2ee33e40-dba2-418c-97f0-f120d64a81f9\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") "
Oct 31 13:53:42.841714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.841699    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-model-cache\") pod \"2ee33e40-dba2-418c-97f0-f120d64a81f9\" (UID: \"2ee33e40-dba2-418c-97f0-f120d64a81f9\") "
Oct 31 13:53:42.841911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.841836    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-home" pod "2ee33e40-dba2-418c-97f0-f120d64a81f9" (UID: "2ee33e40-dba2-418c-97f0-f120d64a81f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:42.841911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.841854    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-tmp-dir" pod "2ee33e40-dba2-418c-97f0-f120d64a81f9" (UID: "2ee33e40-dba2-418c-97f0-f120d64a81f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:42.842145 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.842111    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-model-cache" pod "2ee33e40-dba2-418c-97f0-f120d64a81f9" (UID: "2ee33e40-dba2-418c-97f0-f120d64a81f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:42.842670 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.842646    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:42.842725 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.842678    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:42.842725 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.842694    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:42.844003 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.843976    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-dshm" pod "2ee33e40-dba2-418c-97f0-f120d64a81f9" (UID: "2ee33e40-dba2-418c-97f0-f120d64a81f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:42.844104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.844036    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ee33e40-dba2-418c-97f0-f120d64a81f9-tls-certs" pod "2ee33e40-dba2-418c-97f0-f120d64a81f9" (UID: "2ee33e40-dba2-418c-97f0-f120d64a81f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:53:42.844150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.844108    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee33e40-dba2-418c-97f0-f120d64a81f9-kube-api-access-8vmrf" pod "2ee33e40-dba2-418c-97f0-f120d64a81f9" (UID: "2ee33e40-dba2-418c-97f0-f120d64a81f9"). InnerVolumeSpecName "kube-api-access-8vmrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:53:42.943938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.943863    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vmrf\" (UniqueName: \"kubernetes.io/projected/2ee33e40-dba2-418c-97f0-f120d64a81f9-kube-api-access-8vmrf\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:42.943938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.943934    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ee33e40-dba2-418c-97f0-f120d64a81f9-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:42.943938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:42.943944    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ee33e40-dba2-418c-97f0-f120d64a81f9-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:43.131662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:43.131429    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:43.337710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:43.337670    4428 generic.go:356] "Generic (PLEG): container finished" podID="2ee33e40-dba2-418c-97f0-f120d64a81f9" containerID="0345a1a15b79c25cc4f30462ccf6cb6c9cc577cf362327dfc387a392c0a38f6d" exitCode=0
Oct 31 13:53:43.337978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:43.337724    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q" event={"ID":"2ee33e40-dba2-418c-97f0-f120d64a81f9","Type":"ContainerDied","Data":"0345a1a15b79c25cc4f30462ccf6cb6c9cc577cf362327dfc387a392c0a38f6d"}
Oct 31 13:53:43.337978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:43.337731    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"
Oct 31 13:53:43.337978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:43.337756    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q" event={"ID":"2ee33e40-dba2-418c-97f0-f120d64a81f9","Type":"ContainerDied","Data":"c164131d519c9c53e4c836f56d8d17ca3ca5c4c7e2c166aa74d091f73b7de5f4"}
Oct 31 13:53:43.337978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:43.337772    4428 scope.go:122] "RemoveContainer" containerID="0345a1a15b79c25cc4f30462ccf6cb6c9cc577cf362327dfc387a392c0a38f6d"
Oct 31 13:53:43.350023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:43.350002    4428 scope.go:122] "RemoveContainer" containerID="0345a1a15b79c25cc4f30462ccf6cb6c9cc577cf362327dfc387a392c0a38f6d"
Oct 31 13:53:43.350321 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:53:43.350303    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0345a1a15b79c25cc4f30462ccf6cb6c9cc577cf362327dfc387a392c0a38f6d\": container with ID starting with 0345a1a15b79c25cc4f30462ccf6cb6c9cc577cf362327dfc387a392c0a38f6d not found: ID does not exist" containerID="0345a1a15b79c25cc4f30462ccf6cb6c9cc577cf362327dfc387a392c0a38f6d"
Oct 31 13:53:43.350367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:43.350329    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0345a1a15b79c25cc4f30462ccf6cb6c9cc577cf362327dfc387a392c0a38f6d"} err="failed to get container status \"0345a1a15b79c25cc4f30462ccf6cb6c9cc577cf362327dfc387a392c0a38f6d\": rpc error: code = NotFound desc = could not find container \"0345a1a15b79c25cc4f30462ccf6cb6c9cc577cf362327dfc387a392c0a38f6d\": container with ID starting with 0345a1a15b79c25cc4f30462ccf6cb6c9cc577cf362327dfc387a392c0a38f6d not found: ID does not exist"
Oct 31 13:53:43.369604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:43.369570    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"]
Oct 31 13:53:43.373708 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:43.373683    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-15dc2f/auth-test-model-15dc2f-kserve-57b7c69bf7-gg75q"]
Oct 31 13:53:43.994900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:43.993919    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:44.468236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.468187    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68" podUID="a034fcda-0a8c-44b7-ab07-4e36f9520135" containerName="main" containerID="cri-o://c6d0b3ec135567f265530384039ce81ad7eaad2899bc5a3abeabb086fb95b963" gracePeriod=45
Oct 31 13:53:44.758733 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.758703    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ee33e40-dba2-418c-97f0-f120d64a81f9" path="/var/lib/kubelet/pods/2ee33e40-dba2-418c-97f0-f120d64a81f9/volumes"
Oct 31 13:53:44.767536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.767511    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:53:44.887108 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.887067    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-tmp-dir\") pod \"a034fcda-0a8c-44b7-ab07-4e36f9520135\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") "
Oct 31 13:53:44.887503 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.887473    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-home\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-home\") pod \"a034fcda-0a8c-44b7-ab07-4e36f9520135\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") "
Oct 31 13:53:44.887802 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.887700    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-dshm\") pod \"a034fcda-0a8c-44b7-ab07-4e36f9520135\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") "
Oct 31 13:53:44.888140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.888119    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a034fcda-0a8c-44b7-ab07-4e36f9520135-tls-certs\" (UniqueName: \"kubernetes.io/secret/a034fcda-0a8c-44b7-ab07-4e36f9520135-tls-certs\") pod \"a034fcda-0a8c-44b7-ab07-4e36f9520135\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") "
Oct 31 13:53:44.888309 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.888289    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-model-cache\") pod \"a034fcda-0a8c-44b7-ab07-4e36f9520135\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") "
Oct 31 13:53:44.888454 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.888421    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a034fcda-0a8c-44b7-ab07-4e36f9520135-kube-api-access-zzchx\" (UniqueName: \"kubernetes.io/projected/a034fcda-0a8c-44b7-ab07-4e36f9520135-kube-api-access-zzchx\") pod \"a034fcda-0a8c-44b7-ab07-4e36f9520135\" (UID: \"a034fcda-0a8c-44b7-ab07-4e36f9520135\") "
Oct 31 13:53:44.916894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.907494    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:44.916894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.910157    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-model-cache" pod "a034fcda-0a8c-44b7-ab07-4e36f9520135" (UID: "a034fcda-0a8c-44b7-ab07-4e36f9520135"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:44.916894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.911394    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a034fcda-0a8c-44b7-ab07-4e36f9520135-tls-certs" pod "a034fcda-0a8c-44b7-ab07-4e36f9520135" (UID: "a034fcda-0a8c-44b7-ab07-4e36f9520135"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:53:44.927135 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.921168    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-home" pod "a034fcda-0a8c-44b7-ab07-4e36f9520135" (UID: "a034fcda-0a8c-44b7-ab07-4e36f9520135"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:44.927135 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.922380    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-tmp-dir" pod "a034fcda-0a8c-44b7-ab07-4e36f9520135" (UID: "a034fcda-0a8c-44b7-ab07-4e36f9520135"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:44.927135 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.924704    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-dshm" pod "a034fcda-0a8c-44b7-ab07-4e36f9520135" (UID: "a034fcda-0a8c-44b7-ab07-4e36f9520135"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:53:44.927135 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.925791    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a034fcda-0a8c-44b7-ab07-4e36f9520135-kube-api-access-zzchx" pod "a034fcda-0a8c-44b7-ab07-4e36f9520135" (UID: "a034fcda-0a8c-44b7-ab07-4e36f9520135"). InnerVolumeSpecName "kube-api-access-zzchx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:53:44.990746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.990654    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:44.990746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.990686    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:44.990746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.990698    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a034fcda-0a8c-44b7-ab07-4e36f9520135-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:44.990746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.990707    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:44.990746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.990717    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzchx\" (UniqueName: \"kubernetes.io/projected/a034fcda-0a8c-44b7-ab07-4e36f9520135-kube-api-access-zzchx\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:44.990746 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:44.990729    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a034fcda-0a8c-44b7-ab07-4e36f9520135-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:53:45.354923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:45.354810    4428 generic.go:356] "Generic (PLEG): container finished" podID="a034fcda-0a8c-44b7-ab07-4e36f9520135" containerID="c6d0b3ec135567f265530384039ce81ad7eaad2899bc5a3abeabb086fb95b963" exitCode=0
Oct 31 13:53:45.354923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:45.354852    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"
Oct 31 13:53:45.354923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:45.354909    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68" event={"ID":"a034fcda-0a8c-44b7-ab07-4e36f9520135","Type":"ContainerDied","Data":"c6d0b3ec135567f265530384039ce81ad7eaad2899bc5a3abeabb086fb95b963"}
Oct 31 13:53:45.355514 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:45.354964    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68" event={"ID":"a034fcda-0a8c-44b7-ab07-4e36f9520135","Type":"ContainerDied","Data":"84764ef6663fc38dc7c2d370f25aef0fc0dd8d13817b5b97e9043434e017e85f"}
Oct 31 13:53:45.355514 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:45.354983    4428 scope.go:122] "RemoveContainer" containerID="c6d0b3ec135567f265530384039ce81ad7eaad2899bc5a3abeabb086fb95b963"
Oct 31 13:53:45.366019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:45.365999    4428 scope.go:122] "RemoveContainer" containerID="c6d0b3ec135567f265530384039ce81ad7eaad2899bc5a3abeabb086fb95b963"
Oct 31 13:53:45.366319 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:53:45.366300    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d0b3ec135567f265530384039ce81ad7eaad2899bc5a3abeabb086fb95b963\": container with ID starting with c6d0b3ec135567f265530384039ce81ad7eaad2899bc5a3abeabb086fb95b963 not found: ID does not exist" containerID="c6d0b3ec135567f265530384039ce81ad7eaad2899bc5a3abeabb086fb95b963"
Oct 31 13:53:45.366394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:45.366327    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d0b3ec135567f265530384039ce81ad7eaad2899bc5a3abeabb086fb95b963"} err="failed to get container status \"c6d0b3ec135567f265530384039ce81ad7eaad2899bc5a3abeabb086fb95b963\": rpc error: code = NotFound desc = could not find container \"c6d0b3ec135567f265530384039ce81ad7eaad2899bc5a3abeabb086fb95b963\": container with ID starting with c6d0b3ec135567f265530384039ce81ad7eaad2899bc5a3abeabb086fb95b963 not found: ID does not exist"
Oct 31 13:53:45.389732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:45.389694    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"]
Oct 31 13:53:45.394656 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:45.394613    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-5e8f29/auth-test-model-5e8f29-kserve-5ccf44b869-vrp68"]
Oct 31 13:53:45.699077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:45.698733    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:46.598664 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:46.598619    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:46.756786 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:46.756752    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a034fcda-0a8c-44b7-ab07-4e36f9520135" path="/var/lib/kubelet/pods/a034fcda-0a8c-44b7-ab07-4e36f9520135/volumes"
Oct 31 13:53:47.422895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:47.413767    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:48.295896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:48.288388    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:48.743102 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:53:48.743061    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:53:49.041601 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:49.040483    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:49.829617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:49.827083    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:50.498649 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.498612    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"]
Oct 31 13:53:50.499889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.499843    4428 cpu_manager.go:414] "containerMap: removing container" podUID="6002eb05-9237-48f2-972a-448248703ca4" containerName="istio-proxy"
Oct 31 13:53:50.499889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.499888    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6002eb05-9237-48f2-972a-448248703ca4" containerName="istio-proxy"
Oct 31 13:53:50.500079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.499932    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9ec48f74-8d2b-4680-8e00-1ccd2ac03c32" containerName="test-curl-11125-b2d66a"
Oct 31 13:53:50.500079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.499940    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec48f74-8d2b-4680-8e00-1ccd2ac03c32" containerName="test-curl-11125-b2d66a"
Oct 31 13:53:50.500079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.499956    4428 cpu_manager.go:414] "containerMap: removing container" podUID="6002eb05-9237-48f2-972a-448248703ca4" containerName="istio-proxy"
Oct 31 13:53:50.500079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.499964    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6002eb05-9237-48f2-972a-448248703ca4" containerName="istio-proxy"
Oct 31 13:53:50.500079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.499983    4428 cpu_manager.go:414] "containerMap: removing container" podUID="38ef9813-0afd-477c-8e9f-1bcbfbfd17a4" containerName="main"
Oct 31 13:53:50.500079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.499990    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ef9813-0afd-477c-8e9f-1bcbfbfd17a4" containerName="main"
Oct 31 13:53:50.500079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500026    4428 cpu_manager.go:414] "containerMap: removing container" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy"
Oct 31 13:53:50.500079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500033    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy"
Oct 31 13:53:50.500079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500050    4428 cpu_manager.go:414] "containerMap: removing container" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy"
Oct 31 13:53:50.500079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500059    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy"
Oct 31 13:53:50.500079 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500076    4428 cpu_manager.go:414] "containerMap: removing container" podUID="208fddf2-00ef-4f4a-944a-f3d93ba37911" containerName="istio-proxy"
Oct 31 13:53:50.500441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500083    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208fddf2-00ef-4f4a-944a-f3d93ba37911" containerName="istio-proxy"
Oct 31 13:53:50.500441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500107    4428 cpu_manager.go:414] "containerMap: removing container" podUID="2ee33e40-dba2-418c-97f0-f120d64a81f9" containerName="main"
Oct 31 13:53:50.500441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500114    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ee33e40-dba2-418c-97f0-f120d64a81f9" containerName="main"
Oct 31 13:53:50.500441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500136    4428 cpu_manager.go:414] "containerMap: removing container" podUID="a034fcda-0a8c-44b7-ab07-4e36f9520135" containerName="main"
Oct 31 13:53:50.500441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500143    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a034fcda-0a8c-44b7-ab07-4e36f9520135" containerName="main"
Oct 31 13:53:50.500441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500326    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6002eb05-9237-48f2-972a-448248703ca4" containerName="istio-proxy"
Oct 31 13:53:50.500441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500343    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec48f74-8d2b-4680-8e00-1ccd2ac03c32" containerName="test-curl-11125-b2d66a"
Oct 31 13:53:50.500441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500354    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ef9813-0afd-477c-8e9f-1bcbfbfd17a4" containerName="main"
Oct 31 13:53:50.500441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500367    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy"
Oct 31 13:53:50.500441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500376    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1667cc7b-35eb-4c92-9fa7-239aaa5990d1" containerName="istio-proxy"
Oct 31 13:53:50.500441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500385    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="208fddf2-00ef-4f4a-944a-f3d93ba37911" containerName="istio-proxy"
Oct 31 13:53:50.500441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500397    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ee33e40-dba2-418c-97f0-f120d64a81f9" containerName="main"
Oct 31 13:53:50.500441 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.500406    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a034fcda-0a8c-44b7-ab07-4e36f9520135" containerName="main"
Oct 31 13:53:50.509649 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.509616    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:53:50.517187 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.517154    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4be675-serving-cert\""
Oct 31 13:53:50.522463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.522424    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"]
Oct 31 13:53:50.609640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.609603    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6f1d83-780e-4779-a9b6-fe0b967f149a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf\" (UID: \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:53:50.609851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.609655    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6f1d83-780e-4779-a9b6-fe0b967f149a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf\" (UID: \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:53:50.609851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.609773    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb2jv\" (UniqueName: \"kubernetes.io/projected/2e6f1d83-780e-4779-a9b6-fe0b967f149a-kube-api-access-gb2jv\") pod \"maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf\" (UID: \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:53:50.609993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.609935    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6f1d83-780e-4779-a9b6-fe0b967f149a-metrics-certs\") pod \"maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf\" (UID: \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:53:50.667211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.667164    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:50.720892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.712659    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb2jv\" (UniqueName: \"kubernetes.io/projected/2e6f1d83-780e-4779-a9b6-fe0b967f149a-kube-api-access-gb2jv\") pod \"maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf\" (UID: \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:53:50.720892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.712860    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6f1d83-780e-4779-a9b6-fe0b967f149a-metrics-certs\") pod \"maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf\" (UID: \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:53:50.720892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.713068    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6f1d83-780e-4779-a9b6-fe0b967f149a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf\" (UID: \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:53:50.720892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.713169    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6f1d83-780e-4779-a9b6-fe0b967f149a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf\" (UID: \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:53:50.720892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.714370    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6f1d83-780e-4779-a9b6-fe0b967f149a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf\" (UID: \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:53:50.752976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.744727    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6f1d83-780e-4779-a9b6-fe0b967f149a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf\" (UID: \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:53:50.760568 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.756669    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6f1d83-780e-4779-a9b6-fe0b967f149a-metrics-certs\") pod \"maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf\" (UID: \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:53:50.790421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.790372    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb2jv\" (UniqueName: \"kubernetes.io/projected/2e6f1d83-780e-4779-a9b6-fe0b967f149a-kube-api-access-gb2jv\") pod \"maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf\" (UID: \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:53:50.824755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.824728    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:50.841142 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:50.841120    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:53:51.188717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:51.188687    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"]
Oct 31 13:53:51.190711 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:53:51.190673    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6f1d83_780e_4779_a9b6_fe0b967f149a.slice/crio-7a657980fcd90c334ea90c724ab8af610b30a0e2a516ace7065739b41df93ec1 WatchSource:0}: Error finding container 7a657980fcd90c334ea90c724ab8af610b30a0e2a516ace7065739b41df93ec1: Status 404 returned error can't find the container with id 7a657980fcd90c334ea90c724ab8af610b30a0e2a516ace7065739b41df93ec1
Oct 31 13:53:51.401711 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:51.401672    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf" event={"ID":"2e6f1d83-780e-4779-a9b6-fe0b967f149a","Type":"ContainerStarted","Data":"7a657980fcd90c334ea90c724ab8af610b30a0e2a516ace7065739b41df93ec1"}
Oct 31 13:53:51.515917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:51.515456    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:52.413894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:52.410593    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:52.440790 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:52.440752    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf" event={"ID":"2e6f1d83-780e-4779-a9b6-fe0b967f149a","Type":"ContainerStarted","Data":"9ca6c25f7a29c98f7a03c3ef196b598274c938cffb86c4fe062d099eabbe726e"}
Oct 31 13:53:52.441013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:52.440990    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:53:52.464515 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:52.464461    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf" podStartSLOduration=1.999454369 podStartE2EDuration="2.464445681s" totalImagesPullingTime="464.991312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:52.462666448 +0000 UTC m=+3029.861015073" watchObservedRunningTime="2026-10-31 13:53:52.464445681 +0000 UTC m=+3029.862794304"
Oct 31 13:53:53.311931 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:53.305305    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:54.082403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:54.082363    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:54.880718 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:54.880685    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"]
Oct 31 13:53:54.881687 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:54.881664    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6002eb05-9237-48f2-972a-448248703ca4" containerName="istio-proxy"
Oct 31 13:53:54.885304 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:54.885280    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 13:53:54.905959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:54.903029    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a81979-serving-cert\""
Oct 31 13:53:54.905959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:54.905671    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"]
Oct 31 13:53:54.914927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:54.912135    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4wvl\" (UniqueName: \"kubernetes.io/projected/5c65ac2b-9ff4-4752-8325-bb8bd027708f-kube-api-access-z4wvl\") pod \"maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq\" (UID: \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 13:53:54.914927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:54.912298    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c65ac2b-9ff4-4752-8325-bb8bd027708f-metrics-certs\") pod \"maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq\" (UID: \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 13:53:54.914927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:54.912431    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c65ac2b-9ff4-4752-8325-bb8bd027708f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq\" (UID: \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 13:53:54.914927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:54.912491    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c65ac2b-9ff4-4752-8325-bb8bd027708f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq\" (UID: \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 13:53:54.997426 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:54.997387    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:55.040689 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:55.040529    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c65ac2b-9ff4-4752-8325-bb8bd027708f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq\" (UID: \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 13:53:55.041078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:55.041032    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c65ac2b-9ff4-4752-8325-bb8bd027708f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq\" (UID: \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 13:53:55.041482 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:55.041455    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4wvl\" (UniqueName: \"kubernetes.io/projected/5c65ac2b-9ff4-4752-8325-bb8bd027708f-kube-api-access-z4wvl\") pod \"maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq\" (UID: \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 13:53:55.041739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:55.041716    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c65ac2b-9ff4-4752-8325-bb8bd027708f-metrics-certs\") pod \"maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq\" (UID: \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 13:53:55.042364 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:55.042331    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c65ac2b-9ff4-4752-8325-bb8bd027708f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq\" (UID: \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 13:53:55.055002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:55.053726    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c65ac2b-9ff4-4752-8325-bb8bd027708f-metrics-certs\") pod \"maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq\" (UID: \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 13:53:55.059540 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:55.059508    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c65ac2b-9ff4-4752-8325-bb8bd027708f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq\" (UID: \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 13:53:55.061706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:55.061676    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4wvl\" (UniqueName: \"kubernetes.io/projected/5c65ac2b-9ff4-4752-8325-bb8bd027708f-kube-api-access-z4wvl\") pod \"maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq\" (UID: \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 13:53:55.158772 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:55.158677    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:55.217629 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:55.217592    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 13:53:55.587147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:55.586471    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"]
Oct 31 13:53:55.587467 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:53:55.587423    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c65ac2b_9ff4_4752_8325_bb8bd027708f.slice/crio-32dad84755df6206bb1c72abd79549a9599d85d4bd0cf13cc225da78cac1e70b WatchSource:0}: Error finding container 32dad84755df6206bb1c72abd79549a9599d85d4bd0cf13cc225da78cac1e70b: Status 404 returned error can't find the container with id 32dad84755df6206bb1c72abd79549a9599d85d4bd0cf13cc225da78cac1e70b
Oct 31 13:53:56.477762 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:56.477607    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:56.583355 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:56.583320    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq" event={"ID":"5c65ac2b-9ff4-4752-8325-bb8bd027708f","Type":"ContainerStarted","Data":"e1fa52666d1ee9f296e94708571020058a39196843ac1e221bd1b594946fbd9f"}
Oct 31 13:53:56.583563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:56.583385    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq" event={"ID":"5c65ac2b-9ff4-4752-8325-bb8bd027708f","Type":"ContainerStarted","Data":"32dad84755df6206bb1c72abd79549a9599d85d4bd0cf13cc225da78cac1e70b"}
Oct 31 13:53:56.583563 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:56.583442    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 13:53:56.607261 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:56.607203    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq" podStartSLOduration=2.077439398 podStartE2EDuration="2.607187447s" totalImagesPullingTime="529.748049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:56.603971291 +0000 UTC m=+3034.002319928" watchObservedRunningTime="2026-10-31 13:53:56.607187447 +0000 UTC m=+3034.005536070"
Oct 31 13:53:56.659476 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:56.659376    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:57.101169 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:57.101132    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:57.371426 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:57.371327    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:57.661950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:57.656386    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:58.474372 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:58.474338    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:53:58.590148 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:58.590111    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:53:59.516124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:53:59.516086    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:00.388958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:00.387713    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:00.930352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:00.930322    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:01.087427 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:01.087383    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:01.309846 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:01.309080    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:02.174393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.174350    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:02.203511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.203476    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"]
Oct 31 13:54:02.207812 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.207794    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"
Oct 31 13:54:02.213312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.213286    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8a47d6\""
Oct 31 13:54:02.213734 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.213713    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8a47d6-dockercfg-cgxxk\""
Oct 31 13:54:02.225338 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.219844    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"]
Oct 31 13:54:02.399390 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.353491    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"]
Oct 31 13:54:02.431951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.416803    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmqk6\" (UniqueName: \"kubernetes.io/projected/681992da-f296-455e-b29a-bf0d664b562f-kube-api-access-rmqk6\") pod \"payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj\" (UID: \"681992da-f296-455e-b29a-bf0d664b562f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"
Oct 31 13:54:02.447633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.446335    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681992da-f296-455e-b29a-bf0d664b562f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj\" (UID: \"681992da-f296-455e-b29a-bf0d664b562f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"
Oct 31 13:54:02.447633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.433302    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:02.474896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.470273    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"
Oct 31 13:54:02.584146 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.579648    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681992da-f296-455e-b29a-bf0d664b562f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj\" (UID: \"681992da-f296-455e-b29a-bf0d664b562f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"
Oct 31 13:54:02.619059 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.588241    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681992da-f296-455e-b29a-bf0d664b562f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj\" (UID: \"681992da-f296-455e-b29a-bf0d664b562f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"
Oct 31 13:54:02.619382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.611465    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"]
Oct 31 13:54:02.620440 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.619098    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7l5c\" (UniqueName: \"kubernetes.io/projected/50bddf05-21bc-4e7c-bc88-0807b0e566c0-kube-api-access-m7l5c\") pod \"payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss\" (UID: \"50bddf05-21bc-4e7c-bc88-0807b0e566c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"
Oct 31 13:54:02.620440 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.619634    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50bddf05-21bc-4e7c-bc88-0807b0e566c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss\" (UID: \"50bddf05-21bc-4e7c-bc88-0807b0e566c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"
Oct 31 13:54:02.620440 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.619798    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqk6\" (UniqueName: \"kubernetes.io/projected/681992da-f296-455e-b29a-bf0d664b562f-kube-api-access-rmqk6\") pod \"payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj\" (UID: \"681992da-f296-455e-b29a-bf0d664b562f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"
Oct 31 13:54:02.626558 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.626424    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 13:54:02.681912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.680155    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmqk6\" (UniqueName: \"kubernetes.io/projected/681992da-f296-455e-b29a-bf0d664b562f-kube-api-access-rmqk6\") pod \"payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj\" (UID: \"681992da-f296-455e-b29a-bf0d664b562f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"
Oct 31 13:54:02.722685 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.722644    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7l5c\" (UniqueName: \"kubernetes.io/projected/50bddf05-21bc-4e7c-bc88-0807b0e566c0-kube-api-access-m7l5c\") pod \"payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss\" (UID: \"50bddf05-21bc-4e7c-bc88-0807b0e566c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"
Oct 31 13:54:02.722933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.722716    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50bddf05-21bc-4e7c-bc88-0807b0e566c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss\" (UID: \"50bddf05-21bc-4e7c-bc88-0807b0e566c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"
Oct 31 13:54:02.723974 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.723945    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50bddf05-21bc-4e7c-bc88-0807b0e566c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss\" (UID: \"50bddf05-21bc-4e7c-bc88-0807b0e566c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"
Oct 31 13:54:02.736623 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.736594    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7l5c\" (UniqueName: \"kubernetes.io/projected/50bddf05-21bc-4e7c-bc88-0807b0e566c0-kube-api-access-m7l5c\") pod \"payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss\" (UID: \"50bddf05-21bc-4e7c-bc88-0807b0e566c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"
Oct 31 13:54:02.832599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.832564    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"
Oct 31 13:54:02.900212 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:02.900176    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"
Oct 31 13:54:03.126998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:03.126939    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"]
Oct 31 13:54:03.127219 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:54:03.127190    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681992da_f296_455e_b29a_bf0d664b562f.slice/crio-29146cacb75a2befaba73cd46d00271a90e4d865d181a4e2af6392125185efbe WatchSource:0}: Error finding container 29146cacb75a2befaba73cd46d00271a90e4d865d181a4e2af6392125185efbe: Status 404 returned error can't find the container with id 29146cacb75a2befaba73cd46d00271a90e4d865d181a4e2af6392125185efbe
Oct 31 13:54:03.205015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:03.204145    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:03.252296 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:54:03.252264    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50bddf05_21bc_4e7c_bc88_0807b0e566c0.slice/crio-32956c2eb97776cd6a10ccbbcd8681748e50f0f8c2e03e521f4ebced2843fe22 WatchSource:0}: Error finding container 32956c2eb97776cd6a10ccbbcd8681748e50f0f8c2e03e521f4ebced2843fe22: Status 404 returned error can't find the container with id 32956c2eb97776cd6a10ccbbcd8681748e50f0f8c2e03e521f4ebced2843fe22
Oct 31 13:54:03.253014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:03.252982    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"]
Oct 31 13:54:03.649507 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:03.649397    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss" event={"ID":"50bddf05-21bc-4e7c-bc88-0807b0e566c0","Type":"ContainerStarted","Data":"5a751a24b10e33b5e6a473c9e5ba6bb2c0e3d48edfa262496c6c536411092e59"}
Oct 31 13:54:03.649507 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:03.649472    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss" event={"ID":"50bddf05-21bc-4e7c-bc88-0807b0e566c0","Type":"ContainerStarted","Data":"32956c2eb97776cd6a10ccbbcd8681748e50f0f8c2e03e521f4ebced2843fe22"}
Oct 31 13:54:03.649507 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:03.649501    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"
Oct 31 13:54:03.651407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:03.651375    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj" event={"ID":"681992da-f296-455e-b29a-bf0d664b562f","Type":"ContainerStarted","Data":"f786e03ceeed10cd05ef3d65ae424ce0dd7ba51233bc226c59a12cdf5535f5c0"}
Oct 31 13:54:03.651570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:03.651439    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"
Oct 31 13:54:03.651570 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:03.651463    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj" event={"ID":"681992da-f296-455e-b29a-bf0d664b562f","Type":"ContainerStarted","Data":"29146cacb75a2befaba73cd46d00271a90e4d865d181a4e2af6392125185efbe"}
Oct 31 13:54:03.672433 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:03.672377    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss" podStartSLOduration=1.6723632689999999 podStartE2EDuration="1.672363269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:03.668486879 +0000 UTC m=+3041.066835504" watchObservedRunningTime="2026-10-31 13:54:03.672363269 +0000 UTC m=+3041.070711893"
Oct 31 13:54:03.690351 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:03.690279    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj" podStartSLOduration=1.690254401 podStartE2EDuration="1.690254401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:03.686909262 +0000 UTC m=+3041.085257887" watchObservedRunningTime="2026-10-31 13:54:03.690254401 +0000 UTC m=+3041.088603028"
Oct 31 13:54:04.288893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:04.288619    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:05.226506 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:05.226470    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:06.009903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:06.005563    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:06.805302 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:06.798101    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:07.208630 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:07.206707    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": read tcp 10.134.0.2:45322->10.134.0.247:15021: read: connection reset by peer" start-of-body=
Oct 31 13:54:07.208630 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:07.206774    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj" podUID="c27f0595-0a45-4d55-a097-9c94398bd559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": read tcp 10.134.0.2:45322->10.134.0.247:15021: read: connection reset by peer"
Oct 31 13:54:07.685705 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:07.685666    4428 generic.go:356] "Generic (PLEG): container finished" podID="c27f0595-0a45-4d55-a097-9c94398bd559" containerID="0de7216dfc4aadace9a35e1c2f5d8cc98804c5dd1671d9a3ebb28260f3e7a142" exitCode=0
Oct 31 13:54:07.685946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:07.685739    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj" event={"ID":"c27f0595-0a45-4d55-a097-9c94398bd559","Type":"ContainerDied","Data":"0de7216dfc4aadace9a35e1c2f5d8cc98804c5dd1671d9a3ebb28260f3e7a142"}
Oct 31 13:54:07.686226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:07.686213    4428 scope.go:122] "RemoveContainer" containerID="0de7216dfc4aadace9a35e1c2f5d8cc98804c5dd1671d9a3ebb28260f3e7a142"
Oct 31 13:54:07.688218 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:07.688183    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:54:07.688353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:07.688277    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:54:07.688353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:07.688329    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:54:07.788200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:07.788168    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:08.611899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:08.604814    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:08.696916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:08.696845    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj" event={"ID":"c27f0595-0a45-4d55-a097-9c94398bd559","Type":"ContainerStarted","Data":"132e0b242daadc46b57685e527d391097a2496860633032713538647eeeb6a99"}
Oct 31 13:54:09.416898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:09.407598    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:09.510279 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:09.510242    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:54:09.510465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:09.510401    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 31 13:54:09.510465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:09.510446    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj" podUID="c27f0595-0a45-4d55-a097-9c94398bd559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 31 13:54:10.210897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:10.205152    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:10.510389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:10.510218    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 31 13:54:10.510389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:10.510302    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj" podUID="c27f0595-0a45-4d55-a097-9c94398bd559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 31 13:54:11.005996 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:11.002485    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:11.511219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:11.511183    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 31 13:54:11.511638 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:11.511258    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj" podUID="c27f0595-0a45-4d55-a097-9c94398bd559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 31 13:54:11.838950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:11.830639    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:12.510684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:12.510644    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 31 13:54:12.510957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:12.510719    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj" podUID="c27f0595-0a45-4d55-a097-9c94398bd559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 31 13:54:12.622897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:12.620389    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:13.395643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:13.395604    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:13.511138 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:13.511097    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 31 13:54:13.511363 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:13.511169    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj" podUID="c27f0595-0a45-4d55-a097-9c94398bd559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 31 13:54:14.357002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:14.350094    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:14.586762 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:14.586463    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:54:14.587489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:14.587465    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:54:14.588357 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:14.588335    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:54:14.666847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:14.665290    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"
Oct 31 13:54:14.666847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:14.665966    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"
Oct 31 13:54:14.904030 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:14.903995    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:15.179067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:15.178810    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:15.354894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:15.340467    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:15.510893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:15.509851    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:15.762472 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:15.762390    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:16.077422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:16.077332    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:16.843904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:16.841614    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:17.735894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:17.734957    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:18.618026 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:18.617940    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:18.741947 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:54:18.741910    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:54:19.528892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:19.525181    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:20.303728 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:20.303694    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:20.469854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:20.469801    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:20.645810 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:20.645721    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:21.210466 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:21.199766    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:21.997379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:21.997343    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:22.192777 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:22.192741    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:23.005561 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:23.005517    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:23.882491 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:23.882448    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:24.621295 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:24.621045    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:25.452629 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:25.452543    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:26.398993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:26.398200    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:27.144451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:27.144414    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:27.948779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:27.948293    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:28.966892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:28.961312    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:29.824929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:29.814165    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:30.561555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:30.561459    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:31.386682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:31.386569    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:32.984793 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:32.984758    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:33.649386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:33.649345    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:34.591282 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:34.574344    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:35.382953 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:35.382785    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:35.890545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:35.890512    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:36.047240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:36.047204    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:36.617617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:36.617487    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:37.511592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:37.511537    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:38.243473 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:38.243416    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:39.017895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:39.016546    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:39.826475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:39.826318    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:40.356648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:40.356604    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:40.523966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:40.523890    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:41.088894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:41.087824    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:41.840487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:41.840450    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:42.785794 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:42.785760    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:43.600530 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:43.600488    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:44.506048 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:44.505996    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:45.492788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:45.492730    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:46.295024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:46.294981    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:46.717673 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:46.717639    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"]
Oct 31 13:54:46.734470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:46.734432    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"]
Oct 31 13:54:46.848573 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:46.848510    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:47.020950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:47.013435    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:47.217920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:47.215184    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:48.229582 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:48.229529    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:48.882269 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:54:48.882228    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:54:49.840627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:49.840590    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:50.622680 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:50.622594    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:51.520004 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:51.519861    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:51.718533 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:51.718484    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss" podUID="50bddf05-21bc-4e7c-bc88-0807b0e566c0" containerName="payload-processing" containerID="cri-o://5a751a24b10e33b5e6a473c9e5ba6bb2c0e3d48edfa262496c6c536411092e59" gracePeriod=25
Oct 31 13:54:51.735141 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:51.735100    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj" podUID="681992da-f296-455e-b29a-bf0d664b562f" containerName="payload-pre-processing" containerID="cri-o://f786e03ceeed10cd05ef3d65ae424ce0dd7ba51233bc226c59a12cdf5535f5c0" gracePeriod=25
Oct 31 13:54:52.042612 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.041520    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"
Oct 31 13:54:52.046537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.046331    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50bddf05-21bc-4e7c-bc88-0807b0e566c0-kube-api-access-m7l5c\" (UniqueName: \"kubernetes.io/projected/50bddf05-21bc-4e7c-bc88-0807b0e566c0-kube-api-access-m7l5c\") pod \"50bddf05-21bc-4e7c-bc88-0807b0e566c0\" (UID: \"50bddf05-21bc-4e7c-bc88-0807b0e566c0\") "
Oct 31 13:54:52.046537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.046392    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50bddf05-21bc-4e7c-bc88-0807b0e566c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50bddf05-21bc-4e7c-bc88-0807b0e566c0-plugins-config-volume\") pod \"50bddf05-21bc-4e7c-bc88-0807b0e566c0\" (UID: \"50bddf05-21bc-4e7c-bc88-0807b0e566c0\") "
Oct 31 13:54:52.047518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.047269    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50bddf05-21bc-4e7c-bc88-0807b0e566c0-plugins-config-volume" pod "50bddf05-21bc-4e7c-bc88-0807b0e566c0" (UID: "50bddf05-21bc-4e7c-bc88-0807b0e566c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:54:52.049566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.049537    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50bddf05-21bc-4e7c-bc88-0807b0e566c0-kube-api-access-m7l5c" pod "50bddf05-21bc-4e7c-bc88-0807b0e566c0" (UID: "50bddf05-21bc-4e7c-bc88-0807b0e566c0"). InnerVolumeSpecName "kube-api-access-m7l5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:54:52.079113 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.079050    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"
Oct 31 13:54:52.148057 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.148019    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681992da-f296-455e-b29a-bf0d664b562f-kube-api-access-rmqk6\" (UniqueName: \"kubernetes.io/projected/681992da-f296-455e-b29a-bf0d664b562f-kube-api-access-rmqk6\") pod \"681992da-f296-455e-b29a-bf0d664b562f\" (UID: \"681992da-f296-455e-b29a-bf0d664b562f\") "
Oct 31 13:54:52.148277 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.148175    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/681992da-f296-455e-b29a-bf0d664b562f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681992da-f296-455e-b29a-bf0d664b562f-plugins-config-volume\") pod \"681992da-f296-455e-b29a-bf0d664b562f\" (UID: \"681992da-f296-455e-b29a-bf0d664b562f\") "
Oct 31 13:54:52.148672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.148639    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681992da-f296-455e-b29a-bf0d664b562f-plugins-config-volume" pod "681992da-f296-455e-b29a-bf0d664b562f" (UID: "681992da-f296-455e-b29a-bf0d664b562f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:54:52.149015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.148916    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7l5c\" (UniqueName: \"kubernetes.io/projected/50bddf05-21bc-4e7c-bc88-0807b0e566c0-kube-api-access-m7l5c\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:54:52.149015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.148941    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50bddf05-21bc-4e7c-bc88-0807b0e566c0-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:54:52.149015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.148958    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681992da-f296-455e-b29a-bf0d664b562f-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:54:52.150499 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.150470    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681992da-f296-455e-b29a-bf0d664b562f-kube-api-access-rmqk6" pod "681992da-f296-455e-b29a-bf0d664b562f" (UID: "681992da-f296-455e-b29a-bf0d664b562f"). InnerVolumeSpecName "kube-api-access-rmqk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:54:52.184042 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.184007    4428 generic.go:356] "Generic (PLEG): container finished" podID="50bddf05-21bc-4e7c-bc88-0807b0e566c0" containerID="5a751a24b10e33b5e6a473c9e5ba6bb2c0e3d48edfa262496c6c536411092e59" exitCode=0
Oct 31 13:54:52.184267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.184078    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"
Oct 31 13:54:52.184267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.184107    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss" event={"ID":"50bddf05-21bc-4e7c-bc88-0807b0e566c0","Type":"ContainerDied","Data":"5a751a24b10e33b5e6a473c9e5ba6bb2c0e3d48edfa262496c6c536411092e59"}
Oct 31 13:54:52.184267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.184188    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss" event={"ID":"50bddf05-21bc-4e7c-bc88-0807b0e566c0","Type":"ContainerDied","Data":"32956c2eb97776cd6a10ccbbcd8681748e50f0f8c2e03e521f4ebced2843fe22"}
Oct 31 13:54:52.184267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.184228    4428 scope.go:122] "RemoveContainer" containerID="5a751a24b10e33b5e6a473c9e5ba6bb2c0e3d48edfa262496c6c536411092e59"
Oct 31 13:54:52.189140 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.189109    4428 generic.go:356] "Generic (PLEG): container finished" podID="681992da-f296-455e-b29a-bf0d664b562f" containerID="f786e03ceeed10cd05ef3d65ae424ce0dd7ba51233bc226c59a12cdf5535f5c0" exitCode=0
Oct 31 13:54:52.189287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.189195    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"
Oct 31 13:54:52.189287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.189210    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj" event={"ID":"681992da-f296-455e-b29a-bf0d664b562f","Type":"ContainerDied","Data":"f786e03ceeed10cd05ef3d65ae424ce0dd7ba51233bc226c59a12cdf5535f5c0"}
Oct 31 13:54:52.189287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.189261    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj" event={"ID":"681992da-f296-455e-b29a-bf0d664b562f","Type":"ContainerDied","Data":"29146cacb75a2befaba73cd46d00271a90e4d865d181a4e2af6392125185efbe"}
Oct 31 13:54:52.201231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.199625    4428 scope.go:122] "RemoveContainer" containerID="5a751a24b10e33b5e6a473c9e5ba6bb2c0e3d48edfa262496c6c536411092e59"
Oct 31 13:54:52.201231 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:54:52.200317    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a751a24b10e33b5e6a473c9e5ba6bb2c0e3d48edfa262496c6c536411092e59\": container with ID starting with 5a751a24b10e33b5e6a473c9e5ba6bb2c0e3d48edfa262496c6c536411092e59 not found: ID does not exist" containerID="5a751a24b10e33b5e6a473c9e5ba6bb2c0e3d48edfa262496c6c536411092e59"
Oct 31 13:54:52.201231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.200350    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a751a24b10e33b5e6a473c9e5ba6bb2c0e3d48edfa262496c6c536411092e59"} err="failed to get container status \"5a751a24b10e33b5e6a473c9e5ba6bb2c0e3d48edfa262496c6c536411092e59\": rpc error: code = NotFound desc = could not find container \"5a751a24b10e33b5e6a473c9e5ba6bb2c0e3d48edfa262496c6c536411092e59\": container with ID starting with 5a751a24b10e33b5e6a473c9e5ba6bb2c0e3d48edfa262496c6c536411092e59 not found: ID does not exist"
Oct 31 13:54:52.201231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.200371    4428 scope.go:122] "RemoveContainer" containerID="f786e03ceeed10cd05ef3d65ae424ce0dd7ba51233bc226c59a12cdf5535f5c0"
Oct 31 13:54:52.214953 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.214819    4428 scope.go:122] "RemoveContainer" containerID="f786e03ceeed10cd05ef3d65ae424ce0dd7ba51233bc226c59a12cdf5535f5c0"
Oct 31 13:54:52.215288 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:54:52.215233    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f786e03ceeed10cd05ef3d65ae424ce0dd7ba51233bc226c59a12cdf5535f5c0\": container with ID starting with f786e03ceeed10cd05ef3d65ae424ce0dd7ba51233bc226c59a12cdf5535f5c0 not found: ID does not exist" containerID="f786e03ceeed10cd05ef3d65ae424ce0dd7ba51233bc226c59a12cdf5535f5c0"
Oct 31 13:54:52.215288 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.215268    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f786e03ceeed10cd05ef3d65ae424ce0dd7ba51233bc226c59a12cdf5535f5c0"} err="failed to get container status \"f786e03ceeed10cd05ef3d65ae424ce0dd7ba51233bc226c59a12cdf5535f5c0\": rpc error: code = NotFound desc = could not find container \"f786e03ceeed10cd05ef3d65ae424ce0dd7ba51233bc226c59a12cdf5535f5c0\": container with ID starting with f786e03ceeed10cd05ef3d65ae424ce0dd7ba51233bc226c59a12cdf5535f5c0 not found: ID does not exist"
Oct 31 13:54:52.216249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.216225    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"]
Oct 31 13:54:52.224487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.224461    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78d764f4c-mqbss"]
Oct 31 13:54:52.242808 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.242780    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"]
Oct 31 13:54:52.250790 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.250753    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmqk6\" (UniqueName: \"kubernetes.io/projected/681992da-f296-455e-b29a-bf0d664b562f-kube-api-access-rmqk6\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:54:52.253738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.253713    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-5464c779c5-wh6zj"]
Oct 31 13:54:52.374503 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.374392    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:52.538942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.538833    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:52.699123 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.699040    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"]
Oct 31 13:54:52.700002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.699978    4428 cpu_manager.go:414] "containerMap: removing container" podUID="50bddf05-21bc-4e7c-bc88-0807b0e566c0" containerName="payload-processing"
Oct 31 13:54:52.700002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.700001    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50bddf05-21bc-4e7c-bc88-0807b0e566c0" containerName="payload-processing"
Oct 31 13:54:52.700220 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.700049    4428 cpu_manager.go:414] "containerMap: removing container" podUID="681992da-f296-455e-b29a-bf0d664b562f" containerName="payload-pre-processing"
Oct 31 13:54:52.700220 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.700057    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681992da-f296-455e-b29a-bf0d664b562f" containerName="payload-pre-processing"
Oct 31 13:54:52.700333 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.700260    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="681992da-f296-455e-b29a-bf0d664b562f" containerName="payload-pre-processing"
Oct 31 13:54:52.700333 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.700284    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="50bddf05-21bc-4e7c-bc88-0807b0e566c0" containerName="payload-processing"
Oct 31 13:54:52.706017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.705991    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.711519 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.711495    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-dafbce-openshift-default-dockercfg-sbvx8\""
Oct 31 13:54:52.723535 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.723502    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"]
Oct 31 13:54:52.758251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.758005    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-podinfo\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.758251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.758155    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49p98\" (UniqueName: \"kubernetes.io/projected/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-kube-api-access-49p98\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.758251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.758214    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istiod-ca-cert\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.758600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.758289    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-data\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.758600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.758392    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-workload-certs\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.758600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.758426    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-token\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.758600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.758456    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-envoy\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.758600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.758481    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-workload-socket\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.758917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.758605    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-credential-socket\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.764331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.764301    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50bddf05-21bc-4e7c-bc88-0807b0e566c0" path="/var/lib/kubelet/pods/50bddf05-21bc-4e7c-bc88-0807b0e566c0/volumes"
Oct 31 13:54:52.764759 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.764745    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="681992da-f296-455e-b29a-bf0d664b562f" path="/var/lib/kubelet/pods/681992da-f296-455e-b29a-bf0d664b562f/volumes"
Oct 31 13:54:52.850508 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.850273    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:52.891894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.864858    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-podinfo\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.891894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.864951    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49p98\" (UniqueName: \"kubernetes.io/projected/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-kube-api-access-49p98\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.891894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.864993    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istiod-ca-cert\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.891894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.865025    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-data\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.891894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.865153    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-workload-certs\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.891894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.865203    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-token\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.891894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.865240    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-envoy\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.891894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.865269    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-workload-socket\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.891894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.865431    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-credential-socket\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.891894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.865918    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-credential-socket\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.907808 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.897531    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-data\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.907808 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.898278    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-workload-certs\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.907808 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.898541    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-workload-socket\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.907808 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.902677    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istiod-ca-cert\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.946891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.946538    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49p98\" (UniqueName: \"kubernetes.io/projected/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-kube-api-access-49p98\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:52.961059 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:52.955576    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-token\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:53.019890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:53.019275    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-podinfo\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:53.023331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:53.023268    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-envoy\") pod \"e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") " pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:53.029577 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:53.028152    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:53.352346 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:53.352266    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:53.422151 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:53.421907    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"]
Oct 31 13:54:53.428252 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:54:53.428202    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e427f0a_ca46_4bc6_ad3b_9045d6ac9b86.slice/crio-3bec6d07b2ddbeea37017005d03a4387436ae72ad8d2e502469f4a86a2927323 WatchSource:0}: Error finding container 3bec6d07b2ddbeea37017005d03a4387436ae72ad8d2e502469f4a86a2927323: Status 404 returned error can't find the container with id 3bec6d07b2ddbeea37017005d03a4387436ae72ad8d2e502469f4a86a2927323
Oct 31 13:54:53.432390 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:53.432347    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:54:53.432513 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:53.432470    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:54:53.432566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:53.432533    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:54:53.843041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:53.843001    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:54.210159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:54.210127    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td" event={"ID":"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86","Type":"ContainerStarted","Data":"af47517c0baeef3ceae69ac5f69521d0973bbf65792ccb61ce7f17c82ac5d84e"}
Oct 31 13:54:54.210381 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:54.210198    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td" event={"ID":"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86","Type":"ContainerStarted","Data":"3bec6d07b2ddbeea37017005d03a4387436ae72ad8d2e502469f4a86a2927323"}
Oct 31 13:54:54.240562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:54.240506    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td" podStartSLOduration=2.240492297 podStartE2EDuration="2.240492297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:54.237448402 +0000 UTC m=+3091.635797029" watchObservedRunningTime="2026-10-31 13:54:54.240492297 +0000 UTC m=+3091.638840920"
Oct 31 13:54:54.736959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:54.733516    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:55.029652 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:55.029397    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:55.037497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:55.037467    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:55.218118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:55.218078    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:55.219386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:55.219360    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:54:56.009446 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:56.009411    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:56.628423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:56.628374    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:56.842596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:56.842413    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:57.101208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.101169    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"]
Oct 31 13:54:57.106271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.106244    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.117526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.117495    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8a47d6-openshift-default-dockercfg-ss75h\""
Oct 31 13:54:57.120816 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.120718    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"]
Oct 31 13:54:57.129050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.129015    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.130046 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.130020    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"]
Oct 31 13:54:57.148402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.142553    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"]
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.156234    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-envoy\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.156284    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-workload-certs\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.156326    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-data\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.156361    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-token\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.156399    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istiod-ca-cert\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.156440    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-workload-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.156478    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-workload-certs\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.156681    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-data\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.156758    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8j6t\" (UniqueName: \"kubernetes.io/projected/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-kube-api-access-l8j6t\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.156824    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2nhw\" (UniqueName: \"kubernetes.io/projected/db9fd1b4-227c-46b9-973a-5a96f80597d4-kube-api-access-s2nhw\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.156886    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-workload-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.156912    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-credential-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.156949    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-envoy\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.156999    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-podinfo\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.157049    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9fd1b4-227c-46b9-973a-5a96f80597d4-istiod-ca-cert\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.157093    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-podinfo\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.157176    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-token\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.174730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.157215    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-credential-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.258640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.258599    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-token\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.258916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.258645    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-credential-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.258916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.258687    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-envoy\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.258916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.258703    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-workload-certs\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.258916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.258732    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-data\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.258916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.258780    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-token\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.258916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.258819    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istiod-ca-cert\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.258916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.258883    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-workload-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.258916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.258918    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-workload-certs\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.259419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.259018    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-data\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.259419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.259083    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8j6t\" (UniqueName: \"kubernetes.io/projected/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-kube-api-access-l8j6t\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.259419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.259150    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-credential-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.259419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.259161    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-workload-certs\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.259419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.259158    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2nhw\" (UniqueName: \"kubernetes.io/projected/db9fd1b4-227c-46b9-973a-5a96f80597d4-kube-api-access-s2nhw\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.259419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.259232    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-workload-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.259419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.259257    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-credential-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.259419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.259268    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-workload-certs\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.259419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.259298    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-envoy\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.259419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.259358    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-podinfo\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.260011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.259539    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-data\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.260011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.259594    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9fd1b4-227c-46b9-973a-5a96f80597d4-istiod-ca-cert\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.260011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.259649    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-podinfo\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.260011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.259833    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-credential-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.260255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.260042    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-data\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.260255 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.260108    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-workload-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.260381 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.260281    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istiod-ca-cert\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.260458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.260432    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-workload-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.260702 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.260675    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9fd1b4-227c-46b9-973a-5a96f80597d4-istiod-ca-cert\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.261656 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.261629    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-envoy\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.262236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.262212    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-podinfo\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.262326 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.262216    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-envoy\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.262555 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.262538    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-podinfo\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.280628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.280593    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-token\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.280758 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.280683    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-token\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.280838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.280786    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2nhw\" (UniqueName: \"kubernetes.io/projected/db9fd1b4-227c-46b9-973a-5a96f80597d4-kube-api-access-s2nhw\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.281272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.281250    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8j6t\" (UniqueName: \"kubernetes.io/projected/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-kube-api-access-l8j6t\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.433237 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.433152    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:57.483533 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.483497    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:57.569309 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.569275    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:57.737640 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.737605    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"]
Oct 31 13:54:57.738071 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:54:57.737793    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeed598b_fd5d_49a7_9f9b_46b88c29f64a.slice/crio-2abd4703e3b716826b745226d525bc0b39fb1898159b618e8a487c1cfa353a99 WatchSource:0}: Error finding container 2abd4703e3b716826b745226d525bc0b39fb1898159b618e8a487c1cfa353a99: Status 404 returned error can't find the container with id 2abd4703e3b716826b745226d525bc0b39fb1898159b618e8a487c1cfa353a99
Oct 31 13:54:57.741200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.741164    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:54:57.741313 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.741238    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:54:57.741313 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.741276    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:54:57.809696 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.809658    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"]
Oct 31 13:54:57.811191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.811161    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:57.862914 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:54:57.861900    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9fd1b4_227c_46b9_973a_5a96f80597d4.slice/crio-e375d5aa084f15b61a4f3aa565d56df5c68a88a2b29762fd60c34ded87c49f98 WatchSource:0}: Error finding container e375d5aa084f15b61a4f3aa565d56df5c68a88a2b29762fd60c34ded87c49f98: Status 404 returned error can't find the container with id e375d5aa084f15b61a4f3aa565d56df5c68a88a2b29762fd60c34ded87c49f98
Oct 31 13:54:57.870634 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.870597    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:54:57.870920 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.870894    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:54:57.871141 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:57.871119    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:54:58.027560 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.013547    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:58.034341 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.034308    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"]
Oct 31 13:54:58.048017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.047471    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"
Oct 31 13:54:58.062919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.059096    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-dafbce\""
Oct 31 13:54:58.062919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.062375    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"]
Oct 31 13:54:58.071659 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.067765    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-dafbce-dockercfg-t5r4p\""
Oct 31 13:54:58.073338 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.073313    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7b4286-e631-4534-9ed9-43c7a056ec51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn\" (UID: \"fc7b4286-e631-4534-9ed9-43c7a056ec51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"
Oct 31 13:54:58.073524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.073505    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hcdb\" (UniqueName: \"kubernetes.io/projected/fc7b4286-e631-4534-9ed9-43c7a056ec51-kube-api-access-9hcdb\") pod \"payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn\" (UID: \"fc7b4286-e631-4534-9ed9-43c7a056ec51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"
Oct 31 13:54:58.175929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.175316    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7b4286-e631-4534-9ed9-43c7a056ec51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn\" (UID: \"fc7b4286-e631-4534-9ed9-43c7a056ec51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"
Oct 31 13:54:58.175929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.175378    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hcdb\" (UniqueName: \"kubernetes.io/projected/fc7b4286-e631-4534-9ed9-43c7a056ec51-kube-api-access-9hcdb\") pod \"payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn\" (UID: \"fc7b4286-e631-4534-9ed9-43c7a056ec51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"
Oct 31 13:54:58.270133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.203447    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7b4286-e631-4534-9ed9-43c7a056ec51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn\" (UID: \"fc7b4286-e631-4534-9ed9-43c7a056ec51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"
Oct 31 13:54:58.374894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.372143    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:58.481041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.481000    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh" event={"ID":"db9fd1b4-227c-46b9-973a-5a96f80597d4","Type":"ContainerStarted","Data":"2f32233c3ba5018ad4809b167420d7b543f134e71bb39dfab01fbe6b39a4cf5f"}
Oct 31 13:54:58.481346 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.481317    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh" event={"ID":"db9fd1b4-227c-46b9-973a-5a96f80597d4","Type":"ContainerStarted","Data":"e375d5aa084f15b61a4f3aa565d56df5c68a88a2b29762fd60c34ded87c49f98"}
Oct 31 13:54:58.484078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.483998    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:54:58.495945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.495734    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"]
Oct 31 13:54:58.501122 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.500634    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hcdb\" (UniqueName: \"kubernetes.io/projected/fc7b4286-e631-4534-9ed9-43c7a056ec51-kube-api-access-9hcdb\") pod \"payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn\" (UID: \"fc7b4286-e631-4534-9ed9-43c7a056ec51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"
Oct 31 13:54:58.545679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.534753    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46" event={"ID":"eeed598b-fd5d-49a7-9f9b-46b88c29f64a","Type":"ContainerStarted","Data":"a23403d02a88799388f18a586e51dc81b94ed630903c5bb0547a017045ee918b"}
Oct 31 13:54:58.545679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.534809    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46" event={"ID":"eeed598b-fd5d-49a7-9f9b-46b88c29f64a","Type":"ContainerStarted","Data":"2abd4703e3b716826b745226d525bc0b39fb1898159b618e8a487c1cfa353a99"}
Oct 31 13:54:58.545679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.534942    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.558706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.549731    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-35d13e-openshift-default-dockercfg-wwk8w\""
Oct 31 13:54:58.558706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.550217    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"]
Oct 31 13:54:58.562250 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.559848    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:58.578434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.578372    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh" podStartSLOduration=1.578354918 podStartE2EDuration="1.578354918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:58.551648429 +0000 UTC m=+3095.949997050" watchObservedRunningTime="2026-10-31 13:54:58.578354918 +0000 UTC m=+3095.976703541"
Oct 31 13:54:58.606289 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.606238    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46" podStartSLOduration=1.6062239759999999 podStartE2EDuration="1.606223976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:58.598840838 +0000 UTC m=+3095.997189462" watchObservedRunningTime="2026-10-31 13:54:58.606223976 +0000 UTC m=+3096.004572599"
Oct 31 13:54:58.655421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.655378    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istiod-ca-cert\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.655619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.655504    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-credential-socket\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.655687 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.655612    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sm6s\" (UniqueName: \"kubernetes.io/projected/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-kube-api-access-8sm6s\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.655751 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.655727    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-workload-certs\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.655853 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.655834    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-token\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.655998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.655978    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-data\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.656077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.656025    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-workload-socket\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.656128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.656098    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-envoy\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.656185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.656165    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-podinfo\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.686174 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.686136    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"
Oct 31 13:54:58.760579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.760135    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istiod-ca-cert\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.760579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.760261    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-credential-socket\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.760579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.760294    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sm6s\" (UniqueName: \"kubernetes.io/projected/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-kube-api-access-8sm6s\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.760579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.760341    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-workload-certs\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.760579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.760392    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-token\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.760579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.760467    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-data\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.760579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.760499    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-workload-socket\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.760579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.760523    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-envoy\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.760579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.760561    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-podinfo\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.765915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.762550    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-data\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.765915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.762641    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-credential-socket\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.765915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.762802    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-workload-socket\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.765915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.763248    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-workload-certs\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.765915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.763853    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istiod-ca-cert\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.765915 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.765901    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-envoy\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.766617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.766593    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-podinfo\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.793906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.791014    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-token\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.793906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.791559    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sm6s\" (UniqueName: \"kubernetes.io/projected/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-kube-api-access-8sm6s\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.886397 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.886319    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:54:58.990836 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:54:58.990798    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7b4286_e631_4534_9ed9_43c7a056ec51.slice/crio-f295a4e269e04ec0c810823678d821bd4acc43b843d4377b0ca89e0c76438cd3 WatchSource:0}: Error finding container f295a4e269e04ec0c810823678d821bd4acc43b843d4377b0ca89e0c76438cd3: Status 404 returned error can't find the container with id f295a4e269e04ec0c810823678d821bd4acc43b843d4377b0ca89e0c76438cd3
Oct 31 13:54:58.991307 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:58.991284    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"]
Oct 31 13:54:59.082890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:59.081856    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:54:59.192253 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:59.191803    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"]
Oct 31 13:54:59.192253 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:54:59.191838    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd0ff63_3dca_4fc1_b8b1_9bbd7b3438af.slice/crio-e7027e67f2e35e1f4e74e1fe0e3d53b75e9d1e9f8bf6e6ad2e829ca9e5575808 WatchSource:0}: Error finding container e7027e67f2e35e1f4e74e1fe0e3d53b75e9d1e9f8bf6e6ad2e829ca9e5575808: Status 404 returned error can't find the container with id e7027e67f2e35e1f4e74e1fe0e3d53b75e9d1e9f8bf6e6ad2e829ca9e5575808
Oct 31 13:54:59.197654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:59.196705    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:54:59.197654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:59.196812    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:54:59.197654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:59.196855    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:54:59.433996 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:59.433965    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:54:59.435269 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:59.435237    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Oct 31 13:54:59.435384 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:59.435299    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46" podUID="eeed598b-fd5d-49a7-9f9b-46b88c29f64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Oct 31 13:54:59.485966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:59.485931    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body=
Oct 31 13:54:59.486409 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:59.486353    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh" podUID="db9fd1b4-227c-46b9-973a-5a96f80597d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused"
Oct 31 13:54:59.514956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:59.514908    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z" event={"ID":"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af","Type":"ContainerStarted","Data":"e43e45a6926c728257d09f69f9956393b73b4d8cbde1e8db00c064a770633fce"}
Oct 31 13:54:59.515161 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:59.514995    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z" event={"ID":"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af","Type":"ContainerStarted","Data":"e7027e67f2e35e1f4e74e1fe0e3d53b75e9d1e9f8bf6e6ad2e829ca9e5575808"}
Oct 31 13:54:59.518488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:59.518449    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn" event={"ID":"fc7b4286-e631-4534-9ed9-43c7a056ec51","Type":"ContainerStarted","Data":"75c481035a1af0e5801c46a8ff53ceea2dc7a1b4fc94a69e2f7de7287eb406c0"}
Oct 31 13:54:59.518708 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:59.518559    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn" event={"ID":"fc7b4286-e631-4534-9ed9-43c7a056ec51","Type":"ContainerStarted","Data":"f295a4e269e04ec0c810823678d821bd4acc43b843d4377b0ca89e0c76438cd3"}
Oct 31 13:54:59.542375 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:59.542311    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z" podStartSLOduration=1.5422918989999999 podStartE2EDuration="1.542291899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:59.540459464 +0000 UTC m=+3096.938808094" watchObservedRunningTime="2026-10-31 13:54:59.542291899 +0000 UTC m=+3096.940640525"
Oct 31 13:54:59.591000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:59.590934    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn" podStartSLOduration=1.5909134379999998 podStartE2EDuration="1.590913438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:59.57555771 +0000 UTC m=+3096.973906336" watchObservedRunningTime="2026-10-31 13:54:59.590913438 +0000 UTC m=+3096.989262063"
Oct 31 13:54:59.890974 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:54:59.887352    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:55:00.097125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.097087    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:00.438374 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.436438    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Oct 31 13:55:00.438678 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.438644    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46" podUID="eeed598b-fd5d-49a7-9f9b-46b88c29f64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Oct 31 13:55:00.496895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.486302    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body=
Oct 31 13:55:00.496895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.486366    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh" podUID="db9fd1b4-227c-46b9-973a-5a96f80597d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused"
Oct 31 13:55:00.538673 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.538646    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"
Oct 31 13:55:00.700030 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.699934    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"]
Oct 31 13:55:00.820907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.809134    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.823319 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.822295    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a81979-openshift-default-dockercfg-bg46j\""
Oct 31 13:55:00.898684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.892291    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.898684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.892357    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.898684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.892430    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.898684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.892512    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.898684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.892615    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.898684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.892644    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gwx2\" (UniqueName: \"kubernetes.io/projected/4c0d42c0-825f-4d1f-9670-e695dfe168b1-kube-api-access-2gwx2\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.898684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.892694    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.898684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.892727    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.898684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.892764    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.923898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.913252    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"]
Oct 31 13:55:00.931274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.930021    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 31 13:55:00.931274 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.930102    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z" podUID="4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 31 13:55:00.995896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.995074    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.995896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.995150    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.995896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.995224    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.995896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.995311    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.995896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.995423    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.995896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.995459    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gwx2\" (UniqueName: \"kubernetes.io/projected/4c0d42c0-825f-4d1f-9670-e695dfe168b1-kube-api-access-2gwx2\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.995896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.995510    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.995896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.995546    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:00.995896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.995585    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:01.021921 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:00.996709    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:01.023267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:01.022584    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:01.023480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:01.022826    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:01.024253 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:01.024224    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:01.024978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:01.024955    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:01.027693 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:01.027669    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:01.027856 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:01.027816    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:01.037929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:01.037903    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gwx2\" (UniqueName: \"kubernetes.io/projected/4c0d42c0-825f-4d1f-9670-e695dfe168b1-kube-api-access-2gwx2\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:01.039914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:01.039886    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:01.241080 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:01.241040    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:01.292590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:01.292492    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:01.435894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:01.433827    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Oct 31 13:55:01.435894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:01.433940    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46" podUID="eeed598b-fd5d-49a7-9f9b-46b88c29f64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Oct 31 13:55:01.525960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:01.519070    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body=
Oct 31 13:55:01.525960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:01.519158    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh" podUID="db9fd1b4-227c-46b9-973a-5a96f80597d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused"
Oct 31 13:55:01.960552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:01.960521    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:01.981247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:01.981189    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 31 13:55:01.981618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:01.981582    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z" podUID="4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 31 13:55:02.134337 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:02.134297    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:02.434861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:02.434815    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Oct 31 13:55:02.435292 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:02.435246    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46" podUID="eeed598b-fd5d-49a7-9f9b-46b88c29f64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Oct 31 13:55:02.503515 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:02.503481    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body=
Oct 31 13:55:02.503787 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:02.503761    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh" podUID="db9fd1b4-227c-46b9-973a-5a96f80597d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused"
Oct 31 13:55:02.596291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:02.594413    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:55:02.596291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:02.594516    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:55:02.596291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:02.594560    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:55:02.622247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:02.620702    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"]
Oct 31 13:55:02.725923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:02.725861    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk" event={"ID":"4c0d42c0-825f-4d1f-9670-e695dfe168b1","Type":"ContainerStarted","Data":"1be225a9a309736f38834ef08924474b694d0a165b847422fee410e9e390311d"}
Oct 31 13:55:02.908600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:02.908562    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 31 13:55:02.908912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:02.908881    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z" podUID="4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 31 13:55:02.931243 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:02.931206    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:03.445895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:03.439921    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Oct 31 13:55:03.445895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:03.439998    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46" podUID="eeed598b-fd5d-49a7-9f9b-46b88c29f64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Oct 31 13:55:03.486339 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:03.486302    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body=
Oct 31 13:55:03.486607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:03.486580    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh" podUID="db9fd1b4-227c-46b9-973a-5a96f80597d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused"
Oct 31 13:55:03.751928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:03.751827    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:03.926892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:03.924428    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 31 13:55:03.926892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:03.924490    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z" podUID="4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 31 13:55:04.214129 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:04.204307    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:04.453288 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:04.451996    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Oct 31 13:55:04.453288 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:04.452060    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46" podUID="eeed598b-fd5d-49a7-9f9b-46b88c29f64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Oct 31 13:55:04.497600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:04.492026    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body=
Oct 31 13:55:04.497600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:04.492110    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh" podUID="db9fd1b4-227c-46b9-973a-5a96f80597d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused"
Oct 31 13:55:04.574891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:04.571618    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:04.661287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:04.661229    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk" event={"ID":"4c0d42c0-825f-4d1f-9670-e695dfe168b1","Type":"ContainerStarted","Data":"c278b0991f0b458376a8e14a9790afa4f412e40ba983eac01fa1a78b83c7edcb"}
Oct 31 13:55:04.938259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:04.938031    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 31 13:55:04.938259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:04.938111    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z" podUID="4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 31 13:55:04.941312 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:04.941253    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk" podStartSLOduration=4.941233673 podStartE2EDuration="4.941233673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:04.846587419 +0000 UTC m=+3102.244936054" watchObservedRunningTime="2026-10-31 13:55:04.941233673 +0000 UTC m=+3102.339582298"
Oct 31 13:55:05.241458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:05.241423    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:05.242538 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:05.242517    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Oct 31 13:55:05.242596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:05.242566    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk" podUID="4c0d42c0-825f-4d1f-9670-e695dfe168b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Oct 31 13:55:05.384894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:05.384339    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:05.466894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:05.466644    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Oct 31 13:55:05.466894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:05.466727    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46" podUID="eeed598b-fd5d-49a7-9f9b-46b88c29f64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Oct 31 13:55:05.502143 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:05.502049    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body=
Oct 31 13:55:05.502404 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:05.502375    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh" podUID="db9fd1b4-227c-46b9-973a-5a96f80597d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused"
Oct 31 13:55:05.905957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:05.890457    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 31 13:55:05.905957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:05.890525    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z" podUID="4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 31 13:55:06.242051 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:06.242003    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Oct 31 13:55:06.242288 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:06.242066    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk" podUID="4c0d42c0-825f-4d1f-9670-e695dfe168b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Oct 31 13:55:06.316972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:06.316847    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:06.455151 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:06.455111    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:55:06.456725 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:06.456554    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:55:06.472997 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:06.462277    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:55:06.839887 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:06.839839    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:55:06.840328 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:06.840312    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:55:06.843819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:06.843792    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:55:06.893411 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:06.892962    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 31 13:55:06.893411 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:06.893023    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z" podUID="4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 31 13:55:07.247133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:07.247096    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Oct 31 13:55:07.247933 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:07.247900    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk" podUID="4c0d42c0-825f-4d1f-9670-e695dfe168b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Oct 31 13:55:07.248538 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:07.247799    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:07.897245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:07.897169    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:55:07.908677 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:07.908647    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:55:07.911070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:07.911044    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 13:55:08.243318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:08.242891    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Oct 31 13:55:08.243318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:08.242961    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk" podUID="4c0d42c0-825f-4d1f-9670-e695dfe168b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Oct 31 13:55:08.638021 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:08.637913    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:09.242028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:09.241990    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Oct 31 13:55:09.242461 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:09.242050    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk" podUID="4c0d42c0-825f-4d1f-9670-e695dfe168b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Oct 31 13:55:09.509581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:09.509489    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:10.254080 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:10.249762    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Oct 31 13:55:10.256572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:10.256529    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk" podUID="4c0d42c0-825f-4d1f-9670-e695dfe168b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Oct 31 13:55:10.493912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:10.493742    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:11.242271 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:11.242234    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Oct 31 13:55:11.242486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:11.242294    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk" podUID="4c0d42c0-825f-4d1f-9670-e695dfe168b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Oct 31 13:55:11.398487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:11.398343    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:11.624686 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:11.624602    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"
Oct 31 13:55:12.247034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:12.247001    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:12.247403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:12.247375    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:12.248149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:12.248126    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 13:55:12.368404 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:12.368373    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:13.056642 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:13.056611    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:13.224969 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:13.224834    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:13.915894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:13.908474    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:14.796193 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:14.796153    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:14.997249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:14.997215    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:15.706083 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:15.705982    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:15.863779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:15.863746    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:16.624912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:16.617041    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:17.620894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:17.616483    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:18.449945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:18.449906    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:18.624569 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:18.624532    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:18.762013 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:55:18.761960    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:55:19.334117 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:19.334082    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:19.504676 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:19.504638    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:20.208748 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:20.208703    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:21.161181 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:21.161137    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:21.889903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:21.887593    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:22.785467 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:22.785433    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:22.940137 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:22.940102    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:23.908892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:23.899566    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:24.780126 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:24.769468    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:25.467084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:25.467042    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:25.628072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:25.628034    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:26.535776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:26.535735    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:27.325739 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:27.325699    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:28.339755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:28.339712    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:28.503101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:28.503051    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:29.656601 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:29.656564    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:30.950229 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:30.950187    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:31.199953 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:31.184955    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:32.013543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:32.013448    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:32.819899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:32.819191    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:33.713773 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:33.713730    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:34.540252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:34.540212    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:34.730706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:34.730620    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:35.746814 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:35.746772    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:36.425308 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:36.425152    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:37.299892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:37.290362    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:38.008577 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:38.008540    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:38.931895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:38.905488    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:39.705741 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:39.705705    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:40.130104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:40.130033    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:40.827818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:40.827766    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:41.677534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:41.677470    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:42.390700 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:42.390664    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"]
Oct 31 13:55:42.392077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:42.392037    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626" podUID="39928dfc-c155-42d2-8ab0-8a21e0bcb0d4" containerName="maas-api" containerID="cri-o://8828d2ddd74e3321c86a14b4584a1de5e073af5f691bc19053af81b902be13e2" gracePeriod=30
Oct 31 13:55:42.498189 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:42.498117    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:42.651281 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:42.651164    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:42.861014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:42.832814    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:42.962965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:42.949088    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:55:43.004975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.004824    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:43.125207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.125160    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-openshift-service-ca\") pod \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\" (UID: \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\") "
Oct 31 13:55:43.125978 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.125941    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-kube-api-access-9sk6t\" (UniqueName: \"kubernetes.io/projected/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-kube-api-access-9sk6t\") pod \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\" (UID: \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\") "
Oct 31 13:55:43.126626 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.126605    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-maas-api-tls\") pod \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\" (UID: \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\") "
Oct 31 13:55:43.129031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.128988    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-metrics-certs\") pod \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\" (UID: \"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4\") "
Oct 31 13:55:43.137566 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.125852    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-openshift-service-ca" pod "39928dfc-c155-42d2-8ab0-8a21e0bcb0d4" (UID: "39928dfc-c155-42d2-8ab0-8a21e0bcb0d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:55:43.142158 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.142091    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-maas-api-tls" pod "39928dfc-c155-42d2-8ab0-8a21e0bcb0d4" (UID: "39928dfc-c155-42d2-8ab0-8a21e0bcb0d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:55:43.170896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.170682    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:43.170896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.170790    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:43.176380 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.176187    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-metrics-certs" pod "39928dfc-c155-42d2-8ab0-8a21e0bcb0d4" (UID: "39928dfc-c155-42d2-8ab0-8a21e0bcb0d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:55:43.176380 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.176327    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-kube-api-access-9sk6t" pod "39928dfc-c155-42d2-8ab0-8a21e0bcb0d4" (UID: "39928dfc-c155-42d2-8ab0-8a21e0bcb0d4"). InnerVolumeSpecName "kube-api-access-9sk6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:55:43.187711 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.187643    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:43.223950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.223564    4428 generic.go:356] "Generic (PLEG): container finished" podID="39928dfc-c155-42d2-8ab0-8a21e0bcb0d4" containerID="8828d2ddd74e3321c86a14b4584a1de5e073af5f691bc19053af81b902be13e2" exitCode=0
Oct 31 13:55:43.223950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.223660    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"
Oct 31 13:55:43.223950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.223568    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626" event={"ID":"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4","Type":"ContainerDied","Data":"8828d2ddd74e3321c86a14b4584a1de5e073af5f691bc19053af81b902be13e2"}
Oct 31 13:55:43.223950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.223814    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626" event={"ID":"39928dfc-c155-42d2-8ab0-8a21e0bcb0d4","Type":"ContainerDied","Data":"154c1dcda7026099861136fae8e70eddcb8c182f3c085e08154491c5bcd41625"}
Oct 31 13:55:43.223950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.223855    4428 scope.go:122] "RemoveContainer" containerID="8828d2ddd74e3321c86a14b4584a1de5e073af5f691bc19053af81b902be13e2"
Oct 31 13:55:43.245077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.245046    4428 scope.go:122] "RemoveContainer" containerID="8828d2ddd74e3321c86a14b4584a1de5e073af5f691bc19053af81b902be13e2"
Oct 31 13:55:43.254104 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:55:43.245487    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8828d2ddd74e3321c86a14b4584a1de5e073af5f691bc19053af81b902be13e2\": container with ID starting with 8828d2ddd74e3321c86a14b4584a1de5e073af5f691bc19053af81b902be13e2 not found: ID does not exist" containerID="8828d2ddd74e3321c86a14b4584a1de5e073af5f691bc19053af81b902be13e2"
Oct 31 13:55:43.254104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.245531    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8828d2ddd74e3321c86a14b4584a1de5e073af5f691bc19053af81b902be13e2"} err="failed to get container status \"8828d2ddd74e3321c86a14b4584a1de5e073af5f691bc19053af81b902be13e2\": rpc error: code = NotFound desc = could not find container \"8828d2ddd74e3321c86a14b4584a1de5e073af5f691bc19053af81b902be13e2\": container with ID starting with 8828d2ddd74e3321c86a14b4584a1de5e073af5f691bc19053af81b902be13e2 not found: ID does not exist"
Oct 31 13:55:43.272291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.272256    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:43.272291 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.272294    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sk6t\" (UniqueName: \"kubernetes.io/projected/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4-kube-api-access-9sk6t\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:43.280825 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.280795    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"]
Oct 31 13:55:43.285922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.285896    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-20b8a4-549fdfff57-lg626"]
Oct 31 13:55:43.510528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.510440    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:43.608455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.608423    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"]
Oct 31 13:55:43.608699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:43.608674    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf" podUID="ab494491-9445-49af-935d-f33df1c6589b" containerName="istio-proxy" containerID="cri-o://e99ee40f0c47df74e795c282abbaa510b8e995b9259a051b2b95a581f0e81741" gracePeriod=30
Oct 31 13:55:44.190059 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:44.190009    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:44.758682 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:44.758645    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39928dfc-c155-42d2-8ab0-8a21e0bcb0d4" path="/var/lib/kubelet/pods/39928dfc-c155-42d2-8ab0-8a21e0bcb0d4/volumes"
Oct 31 13:55:44.998013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:44.997487    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:45.721226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.721191    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"]
Oct 31 13:55:45.722084 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.722069    4428 cpu_manager.go:414] "containerMap: removing container" podUID="39928dfc-c155-42d2-8ab0-8a21e0bcb0d4" containerName="maas-api"
Oct 31 13:55:45.722147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.722085    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39928dfc-c155-42d2-8ab0-8a21e0bcb0d4" containerName="maas-api"
Oct 31 13:55:45.722223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.722214    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="39928dfc-c155-42d2-8ab0-8a21e0bcb0d4" containerName="maas-api"
Oct 31 13:55:45.727539 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.727517    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.747801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.747766    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"]
Oct 31 13:55:45.762065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.762026    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"]
Oct 31 13:55:45.781241 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.781203    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.818899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.808826    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.819104    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.819221    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.819294    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.819378    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.819408    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89c6w\" (UniqueName: \"kubernetes.io/projected/54b8f412-0415-4f74-9f20-47d392f1cb00-kube-api-access-89c6w\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.829996    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.830107    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.830133    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.830164    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.830263    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.830365    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.830393    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.830483    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.830536    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.830617    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6k74\" (UniqueName: \"kubernetes.io/projected/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-kube-api-access-x6k74\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.831689    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.831756    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b8f412-0415-4f74-9f20-47d392f1cb00-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.837299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.831820    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.860899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.842933    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"]
Oct 31 13:55:45.933244 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.933211    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.933244 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.933251    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.933551 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.933316    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.933551 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.933344    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.933551 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.933395    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6k74\" (UniqueName: \"kubernetes.io/projected/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-kube-api-access-x6k74\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.933551 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.933471    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.933551 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.933519    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b8f412-0415-4f74-9f20-47d392f1cb00-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.933831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.933569    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.933831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.933661    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.933831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.933689    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.933831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.933738    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.933831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.933778    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.933831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.933803    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.933831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.933822    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.934252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.933853    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89c6w\" (UniqueName: \"kubernetes.io/projected/54b8f412-0415-4f74-9f20-47d392f1cb00-kube-api-access-89c6w\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.934252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.933920    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.934252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.933977    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.934252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.934004    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.934252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.934032    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.934252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.934097    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.934252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.934110    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.934670 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.934376    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.934670 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.934635    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.934779 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.934744    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.934976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.934938    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.935364 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.935337    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.935462 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.935400    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.935571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.935547    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b8f412-0415-4f74-9f20-47d392f1cb00-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.937936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.937912    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.938069 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.937987    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.940241 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.940218    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.940323 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.940221    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.947679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.947653    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89c6w\" (UniqueName: \"kubernetes.io/projected/54b8f412-0415-4f74-9f20-47d392f1cb00-kube-api-access-89c6w\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.947788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.947700    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6k74\" (UniqueName: \"kubernetes.io/projected/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-kube-api-access-x6k74\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:45.948168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.948145    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:45.948217 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:45.948146    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:46.065799 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:46.065706    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:46.195460 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:46.195418    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:46.364603 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:46.364570    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"]
Oct 31 13:55:46.365114 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:55:46.365079    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b3d413_e7cb_4505_96b3_d7da1f9e0d85.slice/crio-1abf6ccd8e76f82b518aaf3d9de337366936323ec385490aacf3b723468b0a4f WatchSource:0}: Error finding container 1abf6ccd8e76f82b518aaf3d9de337366936323ec385490aacf3b723468b0a4f: Status 404 returned error can't find the container with id 1abf6ccd8e76f82b518aaf3d9de337366936323ec385490aacf3b723468b0a4f
Oct 31 13:55:46.368747 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:46.368703    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:55:46.368844 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:46.368803    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:55:46.368947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:46.368853    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:55:46.503717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:46.503680    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"]
Oct 31 13:55:46.509437 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:55:46.509406    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b8f412_0415_4f74_9f20_47d392f1cb00.slice/crio-224de854e0149c1d48be58d874d1436b72c1ff8d1acfd17f1cef3a18edbdcbad WatchSource:0}: Error finding container 224de854e0149c1d48be58d874d1436b72c1ff8d1acfd17f1cef3a18edbdcbad: Status 404 returned error can't find the container with id 224de854e0149c1d48be58d874d1436b72c1ff8d1acfd17f1cef3a18edbdcbad
Oct 31 13:55:46.513122 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:46.513088    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:55:46.513223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:46.513158    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:55:46.513223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:46.513188    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:55:46.653235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:46.652739    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:47.282972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:47.282917    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p" event={"ID":"91b3d413-e7cb-4505-96b3-d7da1f9e0d85","Type":"ContainerStarted","Data":"30eceb9873c2cd909fb1fd0a7761ae837f81d0e21f7559ff2f2e77b81f0977e5"}
Oct 31 13:55:47.283498 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:47.282999    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p" event={"ID":"91b3d413-e7cb-4505-96b3-d7da1f9e0d85","Type":"ContainerStarted","Data":"1abf6ccd8e76f82b518aaf3d9de337366936323ec385490aacf3b723468b0a4f"}
Oct 31 13:55:47.285040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:47.285007    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4" event={"ID":"54b8f412-0415-4f74-9f20-47d392f1cb00","Type":"ContainerStarted","Data":"3e232e7c304418539d851d01d4eb7d30f647dd7803a6ee75b61c7ba89a24ff49"}
Oct 31 13:55:47.285197 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:47.285066    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4" event={"ID":"54b8f412-0415-4f74-9f20-47d392f1cb00","Type":"ContainerStarted","Data":"224de854e0149c1d48be58d874d1436b72c1ff8d1acfd17f1cef3a18edbdcbad"}
Oct 31 13:55:47.314821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:47.314771    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p" podStartSLOduration=2.314756506 podStartE2EDuration="2.314756506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:47.311795745 +0000 UTC m=+3144.710144372" watchObservedRunningTime="2026-10-31 13:55:47.314756506 +0000 UTC m=+3144.713105129"
Oct 31 13:55:47.340760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:47.340697    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4" podStartSLOduration=2.340676402 podStartE2EDuration="2.340676402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:47.336877157 +0000 UTC m=+3144.735225809" watchObservedRunningTime="2026-10-31 13:55:47.340676402 +0000 UTC m=+3144.739025025"
Oct 31 13:55:47.427860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:47.427823    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:48.098089 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:48.096619    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:48.098508 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:48.098474    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 31 13:55:48.098801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:48.098741    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p" podUID="91b3d413-e7cb-4505-96b3-d7da1f9e0d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 31 13:55:48.196334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:48.196269    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:48.199086 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:48.199051    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 31 13:55:48.199247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:48.199132    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4" podUID="54b8f412-0415-4f74-9f20-47d392f1cb00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 31 13:55:48.521752 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:48.521721    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:48.997273 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:55:48.997113    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:55:49.100897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.100347    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 31 13:55:49.100897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.100411    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p" podUID="91b3d413-e7cb-4505-96b3-d7da1f9e0d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 31 13:55:49.200521 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.200423    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 31 13:55:49.201306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.200833    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4" podUID="54b8f412-0415-4f74-9f20-47d392f1cb00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 31 13:55:49.567889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.562362    4428 generic.go:356] "Generic (PLEG): container finished" podID="ab494491-9445-49af-935d-f33df1c6589b" containerID="e99ee40f0c47df74e795c282abbaa510b8e995b9259a051b2b95a581f0e81741" exitCode=0
Oct 31 13:55:49.567889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.563398    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf" event={"ID":"ab494491-9445-49af-935d-f33df1c6589b","Type":"ContainerDied","Data":"e99ee40f0c47df74e795c282abbaa510b8e995b9259a051b2b95a581f0e81741"}
Oct 31 13:55:49.568492 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.568058    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:55:49.727900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.724193    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-istio-envoy\") pod \"ab494491-9445-49af-935d-f33df1c6589b\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") "
Oct 31 13:55:49.727900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.724268    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab494491-9445-49af-935d-f33df1c6589b-istio-token\" (UniqueName: \"kubernetes.io/projected/ab494491-9445-49af-935d-f33df1c6589b-istio-token\") pod \"ab494491-9445-49af-935d-f33df1c6589b\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") "
Oct 31 13:55:49.727900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.724319    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-istio-data\") pod \"ab494491-9445-49af-935d-f33df1c6589b\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") "
Oct 31 13:55:49.727900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.724348    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-workload-socket\") pod \"ab494491-9445-49af-935d-f33df1c6589b\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") "
Oct 31 13:55:49.727900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.724401    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab494491-9445-49af-935d-f33df1c6589b-kube-api-access-mw8lk\" (UniqueName: \"kubernetes.io/projected/ab494491-9445-49af-935d-f33df1c6589b-kube-api-access-mw8lk\") pod \"ab494491-9445-49af-935d-f33df1c6589b\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") "
Oct 31 13:55:49.727900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.724441    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab494491-9445-49af-935d-f33df1c6589b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab494491-9445-49af-935d-f33df1c6589b-istio-podinfo\") pod \"ab494491-9445-49af-935d-f33df1c6589b\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") "
Oct 31 13:55:49.727900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.724675    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-credential-socket\") pod \"ab494491-9445-49af-935d-f33df1c6589b\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") "
Oct 31 13:55:49.727900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.724733    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab494491-9445-49af-935d-f33df1c6589b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab494491-9445-49af-935d-f33df1c6589b-istiod-ca-cert\") pod \"ab494491-9445-49af-935d-f33df1c6589b\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") "
Oct 31 13:55:49.727900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.724786    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-workload-certs\") pod \"ab494491-9445-49af-935d-f33df1c6589b\" (UID: \"ab494491-9445-49af-935d-f33df1c6589b\") "
Oct 31 13:55:49.731403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.731189    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-credential-socket" pod "ab494491-9445-49af-935d-f33df1c6589b" (UID: "ab494491-9445-49af-935d-f33df1c6589b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:49.732215 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.731751    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab494491-9445-49af-935d-f33df1c6589b-istiod-ca-cert" pod "ab494491-9445-49af-935d-f33df1c6589b" (UID: "ab494491-9445-49af-935d-f33df1c6589b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:55:49.732215 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.732073    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-workload-socket" pod "ab494491-9445-49af-935d-f33df1c6589b" (UID: "ab494491-9445-49af-935d-f33df1c6589b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:49.732215 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.732097    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-workload-certs" pod "ab494491-9445-49af-935d-f33df1c6589b" (UID: "ab494491-9445-49af-935d-f33df1c6589b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:49.732523 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.732265    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-istio-data" pod "ab494491-9445-49af-935d-f33df1c6589b" (UID: "ab494491-9445-49af-935d-f33df1c6589b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:49.733011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.732985    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-istio-envoy" pod "ab494491-9445-49af-935d-f33df1c6589b" (UID: "ab494491-9445-49af-935d-f33df1c6589b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:49.736125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.736053    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab494491-9445-49af-935d-f33df1c6589b-istio-podinfo" pod "ab494491-9445-49af-935d-f33df1c6589b" (UID: "ab494491-9445-49af-935d-f33df1c6589b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:55:49.736125 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.736101    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab494491-9445-49af-935d-f33df1c6589b-istio-token" pod "ab494491-9445-49af-935d-f33df1c6589b" (UID: "ab494491-9445-49af-935d-f33df1c6589b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:55:49.738646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.738605    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab494491-9445-49af-935d-f33df1c6589b-kube-api-access-mw8lk" pod "ab494491-9445-49af-935d-f33df1c6589b" (UID: "ab494491-9445-49af-935d-f33df1c6589b"). InnerVolumeSpecName "kube-api-access-mw8lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:55:49.827217 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.827181    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:49.827217 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.827218    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:49.827518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.827234    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw8lk\" (UniqueName: \"kubernetes.io/projected/ab494491-9445-49af-935d-f33df1c6589b-kube-api-access-mw8lk\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:49.827518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.827247    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab494491-9445-49af-935d-f33df1c6589b-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:49.827518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.827260    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:49.827518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.827274    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab494491-9445-49af-935d-f33df1c6589b-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:49.827518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.827285    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:49.827518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.827296    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab494491-9445-49af-935d-f33df1c6589b-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:49.827518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.827308    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab494491-9445-49af-935d-f33df1c6589b-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:49.918891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:49.894944    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:50.077145 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:50.076522    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 31 13:55:50.077145 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:50.076610    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p" podUID="91b3d413-e7cb-4505-96b3-d7da1f9e0d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 31 13:55:50.196486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:50.196452    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 31 13:55:50.196722 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:50.196701    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4" podUID="54b8f412-0415-4f74-9f20-47d392f1cb00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 31 13:55:50.411819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:50.411689    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:50.603917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:50.602421    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf" event={"ID":"ab494491-9445-49af-935d-f33df1c6589b","Type":"ContainerDied","Data":"34a8acb05c5ab5ee8fe42d1be1d2b286768f05f987bf37b12ad7d06038489775"}
Oct 31 13:55:50.603917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:50.602455    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"
Oct 31 13:55:50.603917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:50.602528    4428 scope.go:122] "RemoveContainer" containerID="e99ee40f0c47df74e795c282abbaa510b8e995b9259a051b2b95a581f0e81741"
Oct 31 13:55:50.640745 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:50.639762    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"]
Oct 31 13:55:50.712860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:50.691985    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:50.712860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:50.707792    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-7lxzf"]
Oct 31 13:55:50.718190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:50.717443    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"]
Oct 31 13:55:50.718190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:50.717642    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj" podUID="c27f0595-0a45-4d55-a097-9c94398bd559" containerName="istio-proxy" containerID="cri-o://132e0b242daadc46b57685e527d391097a2496860633032713538647eeeb6a99" gracePeriod=30
Oct 31 13:55:51.134947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:51.134658    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 31 13:55:51.134947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:51.134729    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p" podUID="91b3d413-e7cb-4505-96b3-d7da1f9e0d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 31 13:55:51.142590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:51.141565    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:51.173899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:51.160294    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab494491-9445-49af-935d-f33df1c6589b" path="/var/lib/kubelet/pods/ab494491-9445-49af-935d-f33df1c6589b/volumes"
Oct 31 13:55:51.201895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:51.199528    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 31 13:55:51.201895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:51.199595    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4" podUID="54b8f412-0415-4f74-9f20-47d392f1cb00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 31 13:55:51.303899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:51.300505    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:52.068896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:52.068353    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 31 13:55:52.068896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:52.068419    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p" podUID="91b3d413-e7cb-4505-96b3-d7da1f9e0d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 31 13:55:52.196897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:52.196270    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 31 13:55:52.196897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:52.196332    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4" podUID="54b8f412-0415-4f74-9f20-47d392f1cb00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 31 13:55:52.446009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:52.444285    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"]
Oct 31 13:55:52.523260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:52.516738    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:52.924786 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:52.924745    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:53.098899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:53.087531    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 31 13:55:53.098899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:53.087593    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p" podUID="91b3d413-e7cb-4505-96b3-d7da1f9e0d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 31 13:55:53.101327 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:53.101293    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:53.204487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:53.203974    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 31 13:55:53.204487 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:53.204027    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4" podUID="54b8f412-0415-4f74-9f20-47d392f1cb00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 31 13:55:53.961954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:53.961916    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:54.080911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:54.072477    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 31 13:55:54.112014 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:54.111970    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p" podUID="91b3d413-e7cb-4505-96b3-d7da1f9e0d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 31 13:55:54.259968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:54.254370    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body=
Oct 31 13:55:54.259968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:54.254440    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4" podUID="54b8f412-0415-4f74-9f20-47d392f1cb00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused"
Oct 31 13:55:55.090706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:55.090663    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:55.091799 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:55.091774    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:55.096534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:55.096506    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 13:55:55.158895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:55.157053    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:55.243638 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:55.243579    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:55.263896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:55.250247    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:55.289858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:55.285269    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 13:55:56.087383 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.087356    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:55:56.108185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.108152    4428 generic.go:356] "Generic (PLEG): container finished" podID="c27f0595-0a45-4d55-a097-9c94398bd559" containerID="132e0b242daadc46b57685e527d391097a2496860633032713538647eeeb6a99" exitCode=0
Oct 31 13:55:56.108389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.108209    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"
Oct 31 13:55:56.108389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.108245    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj" event={"ID":"c27f0595-0a45-4d55-a097-9c94398bd559","Type":"ContainerDied","Data":"132e0b242daadc46b57685e527d391097a2496860633032713538647eeeb6a99"}
Oct 31 13:55:56.108389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.108302    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj" event={"ID":"c27f0595-0a45-4d55-a097-9c94398bd559","Type":"ContainerDied","Data":"9e583526c3806bed5dbc580a6f7e2c3e22ffe14d6d9969c4ecab76f706739a0a"}
Oct 31 13:55:56.108389 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.108332    4428 scope.go:122] "RemoveContainer" containerID="132e0b242daadc46b57685e527d391097a2496860633032713538647eeeb6a99"
Oct 31 13:55:56.122681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.122662    4428 scope.go:122] "RemoveContainer" containerID="0de7216dfc4aadace9a35e1c2f5d8cc98804c5dd1671d9a3ebb28260f3e7a142"
Oct 31 13:55:56.136451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.136424    4428 scope.go:122] "RemoveContainer" containerID="132e0b242daadc46b57685e527d391097a2496860633032713538647eeeb6a99"
Oct 31 13:55:56.136847 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:55:56.136818    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132e0b242daadc46b57685e527d391097a2496860633032713538647eeeb6a99\": container with ID starting with 132e0b242daadc46b57685e527d391097a2496860633032713538647eeeb6a99 not found: ID does not exist" containerID="132e0b242daadc46b57685e527d391097a2496860633032713538647eeeb6a99"
Oct 31 13:55:56.136949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.136904    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132e0b242daadc46b57685e527d391097a2496860633032713538647eeeb6a99"} err="failed to get container status \"132e0b242daadc46b57685e527d391097a2496860633032713538647eeeb6a99\": rpc error: code = NotFound desc = could not find container \"132e0b242daadc46b57685e527d391097a2496860633032713538647eeeb6a99\": container with ID starting with 132e0b242daadc46b57685e527d391097a2496860633032713538647eeeb6a99 not found: ID does not exist"
Oct 31 13:55:56.136949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.136933    4428 scope.go:122] "RemoveContainer" containerID="0de7216dfc4aadace9a35e1c2f5d8cc98804c5dd1671d9a3ebb28260f3e7a142"
Oct 31 13:55:56.137258 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:55:56.137235    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de7216dfc4aadace9a35e1c2f5d8cc98804c5dd1671d9a3ebb28260f3e7a142\": container with ID starting with 0de7216dfc4aadace9a35e1c2f5d8cc98804c5dd1671d9a3ebb28260f3e7a142 not found: ID does not exist" containerID="0de7216dfc4aadace9a35e1c2f5d8cc98804c5dd1671d9a3ebb28260f3e7a142"
Oct 31 13:55:56.137345 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.137265    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de7216dfc4aadace9a35e1c2f5d8cc98804c5dd1671d9a3ebb28260f3e7a142"} err="failed to get container status \"0de7216dfc4aadace9a35e1c2f5d8cc98804c5dd1671d9a3ebb28260f3e7a142\": rpc error: code = NotFound desc = could not find container \"0de7216dfc4aadace9a35e1c2f5d8cc98804c5dd1671d9a3ebb28260f3e7a142\": container with ID starting with 0de7216dfc4aadace9a35e1c2f5d8cc98804c5dd1671d9a3ebb28260f3e7a142 not found: ID does not exist"
Oct 31 13:55:56.141393 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.141369    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-istio-data\") pod \"c27f0595-0a45-4d55-a097-9c94398bd559\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") "
Oct 31 13:55:56.141536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.141482    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c27f0595-0a45-4d55-a097-9c94398bd559-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27f0595-0a45-4d55-a097-9c94398bd559-istiod-ca-cert\") pod \"c27f0595-0a45-4d55-a097-9c94398bd559\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") "
Oct 31 13:55:56.141596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.141552    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-workload-socket\") pod \"c27f0595-0a45-4d55-a097-9c94398bd559\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") "
Oct 31 13:55:56.141654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.141635    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27f0595-0a45-4d55-a097-9c94398bd559-kube-api-access-rp78t\" (UniqueName: \"kubernetes.io/projected/c27f0595-0a45-4d55-a097-9c94398bd559-kube-api-access-rp78t\") pod \"c27f0595-0a45-4d55-a097-9c94398bd559\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") "
Oct 31 13:55:56.141716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.141674    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c27f0595-0a45-4d55-a097-9c94398bd559-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27f0595-0a45-4d55-a097-9c94398bd559-istio-podinfo\") pod \"c27f0595-0a45-4d55-a097-9c94398bd559\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") "
Oct 31 13:55:56.142168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.141898    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-credential-socket\") pod \"c27f0595-0a45-4d55-a097-9c94398bd559\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") "
Oct 31 13:55:56.142168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.141950    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-workload-certs\") pod \"c27f0595-0a45-4d55-a097-9c94398bd559\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") "
Oct 31 13:55:56.142168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.141955    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-workload-socket" pod "c27f0595-0a45-4d55-a097-9c94398bd559" (UID: "c27f0595-0a45-4d55-a097-9c94398bd559"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:56.142409 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.142169    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27f0595-0a45-4d55-a097-9c94398bd559-istio-token\" (UniqueName: \"kubernetes.io/projected/c27f0595-0a45-4d55-a097-9c94398bd559-istio-token\") pod \"c27f0595-0a45-4d55-a097-9c94398bd559\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") "
Oct 31 13:55:56.142409 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.142244    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-credential-socket" pod "c27f0595-0a45-4d55-a097-9c94398bd559" (UID: "c27f0595-0a45-4d55-a097-9c94398bd559"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:56.142409 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.142280    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-istio-envoy\") pod \"c27f0595-0a45-4d55-a097-9c94398bd559\" (UID: \"c27f0595-0a45-4d55-a097-9c94398bd559\") "
Oct 31 13:55:56.143919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.142640    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-workload-certs" pod "c27f0595-0a45-4d55-a097-9c94398bd559" (UID: "c27f0595-0a45-4d55-a097-9c94398bd559"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:56.143919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.143838    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:56.144168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.143861    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:56.144168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.144134    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:56.144635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.144556    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27f0595-0a45-4d55-a097-9c94398bd559-istiod-ca-cert" pod "c27f0595-0a45-4d55-a097-9c94398bd559" (UID: "c27f0595-0a45-4d55-a097-9c94398bd559"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:55:56.144635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.144622    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-istio-data" pod "c27f0595-0a45-4d55-a097-9c94398bd559" (UID: "c27f0595-0a45-4d55-a097-9c94398bd559"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:56.167477 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.147139    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27f0595-0a45-4d55-a097-9c94398bd559-kube-api-access-rp78t" pod "c27f0595-0a45-4d55-a097-9c94398bd559" (UID: "c27f0595-0a45-4d55-a097-9c94398bd559"). InnerVolumeSpecName "kube-api-access-rp78t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:55:56.167477 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.147241    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c27f0595-0a45-4d55-a097-9c94398bd559-istio-podinfo" pod "c27f0595-0a45-4d55-a097-9c94398bd559" (UID: "c27f0595-0a45-4d55-a097-9c94398bd559"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:55:56.167477 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.149363    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27f0595-0a45-4d55-a097-9c94398bd559-istio-token" pod "c27f0595-0a45-4d55-a097-9c94398bd559" (UID: "c27f0595-0a45-4d55-a097-9c94398bd559"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:55:56.176183 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.175984    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-istio-envoy" pod "c27f0595-0a45-4d55-a097-9c94398bd559" (UID: "c27f0595-0a45-4d55-a097-9c94398bd559"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:55:56.257070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.254783    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27f0595-0a45-4d55-a097-9c94398bd559-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:56.257070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.254823    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:56.257070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.254837    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27f0595-0a45-4d55-a097-9c94398bd559-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:56.257070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.254850    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27f0595-0a45-4d55-a097-9c94398bd559-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:56.257070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.254885    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp78t\" (UniqueName: \"kubernetes.io/projected/c27f0595-0a45-4d55-a097-9c94398bd559-kube-api-access-rp78t\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:56.257070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.254901    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27f0595-0a45-4d55-a097-9c94398bd559-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:56.269053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.258000    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:56.437209 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.437172    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"]
Oct 31 13:55:56.442398 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.442372    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-20b8a4-openshift-default-5cd4d5f945-lt9hj"]
Oct 31 13:55:56.785533 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:56.785501    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c27f0595-0a45-4d55-a097-9c94398bd559" path="/var/lib/kubelet/pods/c27f0595-0a45-4d55-a097-9c94398bd559/volumes"
Oct 31 13:55:57.423020 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:57.416087    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:57.445228 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:57.445184    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn" podUID="fc7b4286-e631-4534-9ed9-43c7a056ec51" containerName="payload-pre-processing" containerID="cri-o://75c481035a1af0e5801c46a8ff53ceea2dc7a1b4fc94a69e2f7de7287eb406c0" gracePeriod=25
Oct 31 13:55:57.840420 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:57.840382    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"
Oct 31 13:55:57.842325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:57.842220    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:57.926530 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:57.926488    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc7b4286-e631-4534-9ed9-43c7a056ec51-kube-api-access-9hcdb\" (UniqueName: \"kubernetes.io/projected/fc7b4286-e631-4534-9ed9-43c7a056ec51-kube-api-access-9hcdb\") pod \"fc7b4286-e631-4534-9ed9-43c7a056ec51\" (UID: \"fc7b4286-e631-4534-9ed9-43c7a056ec51\") "
Oct 31 13:55:57.926794 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:57.926780    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc7b4286-e631-4534-9ed9-43c7a056ec51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7b4286-e631-4534-9ed9-43c7a056ec51-plugins-config-volume\") pod \"fc7b4286-e631-4534-9ed9-43c7a056ec51\" (UID: \"fc7b4286-e631-4534-9ed9-43c7a056ec51\") "
Oct 31 13:55:57.927208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:57.927180    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc7b4286-e631-4534-9ed9-43c7a056ec51-plugins-config-volume" pod "fc7b4286-e631-4534-9ed9-43c7a056ec51" (UID: "fc7b4286-e631-4534-9ed9-43c7a056ec51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:55:57.927403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:57.927384    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7b4286-e631-4534-9ed9-43c7a056ec51-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:57.928906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:57.928862    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7b4286-e631-4534-9ed9-43c7a056ec51-kube-api-access-9hcdb" pod "fc7b4286-e631-4534-9ed9-43c7a056ec51" (UID: "fc7b4286-e631-4534-9ed9-43c7a056ec51"). InnerVolumeSpecName "kube-api-access-9hcdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:55:58.008026 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:58.007988    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:58.029146 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:58.029102    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hcdb\" (UniqueName: \"kubernetes.io/projected/fc7b4286-e631-4534-9ed9-43c7a056ec51-kube-api-access-9hcdb\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:55:58.192946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:58.177064    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:58.210906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:58.195616    4428 generic.go:356] "Generic (PLEG): container finished" podID="fc7b4286-e631-4534-9ed9-43c7a056ec51" containerID="75c481035a1af0e5801c46a8ff53ceea2dc7a1b4fc94a69e2f7de7287eb406c0" exitCode=0
Oct 31 13:55:58.210906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:58.195835    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn" event={"ID":"fc7b4286-e631-4534-9ed9-43c7a056ec51","Type":"ContainerDied","Data":"75c481035a1af0e5801c46a8ff53ceea2dc7a1b4fc94a69e2f7de7287eb406c0"}
Oct 31 13:55:58.210906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:58.195948    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn" event={"ID":"fc7b4286-e631-4534-9ed9-43c7a056ec51","Type":"ContainerDied","Data":"f295a4e269e04ec0c810823678d821bd4acc43b843d4377b0ca89e0c76438cd3"}
Oct 31 13:55:58.210906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:58.195984    4428 scope.go:122] "RemoveContainer" containerID="75c481035a1af0e5801c46a8ff53ceea2dc7a1b4fc94a69e2f7de7287eb406c0"
Oct 31 13:55:58.210906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:58.196093    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"
Oct 31 13:55:58.384619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:58.376456    4428 scope.go:122] "RemoveContainer" containerID="75c481035a1af0e5801c46a8ff53ceea2dc7a1b4fc94a69e2f7de7287eb406c0"
Oct 31 13:55:58.384619 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:55:58.378711    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c481035a1af0e5801c46a8ff53ceea2dc7a1b4fc94a69e2f7de7287eb406c0\": container with ID starting with 75c481035a1af0e5801c46a8ff53ceea2dc7a1b4fc94a69e2f7de7287eb406c0 not found: ID does not exist" containerID="75c481035a1af0e5801c46a8ff53ceea2dc7a1b4fc94a69e2f7de7287eb406c0"
Oct 31 13:55:58.384619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:58.378760    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c481035a1af0e5801c46a8ff53ceea2dc7a1b4fc94a69e2f7de7287eb406c0"} err="failed to get container status \"75c481035a1af0e5801c46a8ff53ceea2dc7a1b4fc94a69e2f7de7287eb406c0\": rpc error: code = NotFound desc = could not find container \"75c481035a1af0e5801c46a8ff53ceea2dc7a1b4fc94a69e2f7de7287eb406c0\": container with ID starting with 75c481035a1af0e5801c46a8ff53ceea2dc7a1b4fc94a69e2f7de7287eb406c0 not found: ID does not exist"
Oct 31 13:55:58.384619 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:58.379346    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"]
Oct 31 13:55:58.388970 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:58.386109    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-79999dc655-cxnbn"]
Oct 31 13:55:58.781836 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:58.781798    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc7b4286-e631-4534-9ed9-43c7a056ec51" path="/var/lib/kubelet/pods/fc7b4286-e631-4534-9ed9-43c7a056ec51/volumes"
Oct 31 13:55:58.929935 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:58.927848    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:59.173976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:59.173835    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:55:59.455950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:55:59.433524    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:00.052898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:00.051605    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:00.308503 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:00.308367    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:01.167898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:01.164932    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:01.766597 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:01.766561    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:02.106051 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:02.104616    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:03.072899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:03.070206    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:04.107067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:04.107027    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:05.055580 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:05.055542    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:05.929909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:05.929628    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:06.276279 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:06.276245    4428 generic.go:356] "Generic (PLEG): container finished" podID="4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" containerID="af47517c0baeef3ceae69ac5f69521d0973bbf65792ccb61ce7f17c82ac5d84e" exitCode=0
Oct 31 13:56:06.276478 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:06.276323    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td" event={"ID":"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86","Type":"ContainerDied","Data":"af47517c0baeef3ceae69ac5f69521d0973bbf65792ccb61ce7f17c82ac5d84e"}
Oct 31 13:56:06.276834 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:06.276820    4428 scope.go:122] "RemoveContainer" containerID="af47517c0baeef3ceae69ac5f69521d0973bbf65792ccb61ce7f17c82ac5d84e"
Oct 31 13:56:06.278814 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:06.278784    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:56:06.278946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:06.278854    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:56:06.278946 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:06.278931    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:56:06.833898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:06.828215    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:07.286265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:07.286217    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td" event={"ID":"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86","Type":"ContainerStarted","Data":"3642a44855090e9f67919838f85632847bace33c7926ae09901524c659c5ab72"}
Oct 31 13:56:07.641686 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:07.641595    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:07.885958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:07.883473    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:08.037571 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:08.037540    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:56:08.042003 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:08.041970    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Oct 31 13:56:08.043013 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:08.042980    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td" podUID="4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Oct 31 13:56:08.174892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:08.165260    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:08.667904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:08.667040    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:08.892608 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:08.892575    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:09.029219 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:09.029181    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Oct 31 13:56:09.029455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:09.029256    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td" podUID="4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Oct 31 13:56:09.855006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:09.854848    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:10.032988 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:10.032947    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Oct 31 13:56:10.033208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:10.033021    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td" podUID="4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Oct 31 13:56:10.117975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:10.112781    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:10.221113 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:10.220039    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:56:10.440270 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:10.440093    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:10.879901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:10.878907    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:11.046929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:11.045313    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Oct 31 13:56:11.046929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:11.045378    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td" podUID="4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Oct 31 13:56:11.088998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:11.088961    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:11.708542 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:11.708507    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:12.046894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:12.039011    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Oct 31 13:56:12.046894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:12.039108    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td" podUID="4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Oct 31 13:56:12.834897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:12.827429    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:13.028764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:13.028729    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body=
Oct 31 13:56:13.028982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:13.028782    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td" podUID="4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused"
Oct 31 13:56:13.973495 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:13.973451    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:14.034788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:14.034754    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:56:14.035800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:14.035778    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:56:14.232942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:14.232835    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:14.396968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:14.396897    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:15.001402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.001364    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:15.164628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.164596    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:15.427971 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.427888    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"]
Oct 31 13:56:15.428765 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.428749    4428 cpu_manager.go:414] "containerMap: removing container" podUID="ab494491-9445-49af-935d-f33df1c6589b" containerName="istio-proxy"
Oct 31 13:56:15.428821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.428767    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab494491-9445-49af-935d-f33df1c6589b" containerName="istio-proxy"
Oct 31 13:56:15.428821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.428782    4428 cpu_manager.go:414] "containerMap: removing container" podUID="c27f0595-0a45-4d55-a097-9c94398bd559" containerName="istio-proxy"
Oct 31 13:56:15.428821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.428788    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27f0595-0a45-4d55-a097-9c94398bd559" containerName="istio-proxy"
Oct 31 13:56:15.428821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.428801    4428 cpu_manager.go:414] "containerMap: removing container" podUID="c27f0595-0a45-4d55-a097-9c94398bd559" containerName="istio-proxy"
Oct 31 13:56:15.428821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.428806    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27f0595-0a45-4d55-a097-9c94398bd559" containerName="istio-proxy"
Oct 31 13:56:15.429019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.428825    4428 cpu_manager.go:414] "containerMap: removing container" podUID="fc7b4286-e631-4534-9ed9-43c7a056ec51" containerName="payload-pre-processing"
Oct 31 13:56:15.429019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.428830    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc7b4286-e631-4534-9ed9-43c7a056ec51" containerName="payload-pre-processing"
Oct 31 13:56:15.429019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.428996    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab494491-9445-49af-935d-f33df1c6589b" containerName="istio-proxy"
Oct 31 13:56:15.429019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.429006    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27f0595-0a45-4d55-a097-9c94398bd559" containerName="istio-proxy"
Oct 31 13:56:15.429019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.429013    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27f0595-0a45-4d55-a097-9c94398bd559" containerName="istio-proxy"
Oct 31 13:56:15.429190 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.429020    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc7b4286-e631-4534-9ed9-43c7a056ec51" containerName="payload-pre-processing"
Oct 31 13:56:15.433249 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.433224    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"
Oct 31 13:56:15.434737 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.434716    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:15.438817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.438793    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e50b22\""
Oct 31 13:56:15.442989 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.442589    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"]
Oct 31 13:56:15.515950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.515916    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2ngp\" (UniqueName: \"kubernetes.io/projected/43c7ca36-3d4d-496f-a3e8-67ec2e9b2243-kube-api-access-v2ngp\") pod \"payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx\" (UID: \"43c7ca36-3d4d-496f-a3e8-67ec2e9b2243\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"
Oct 31 13:56:15.516146 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.516025    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c7ca36-3d4d-496f-a3e8-67ec2e9b2243-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx\" (UID: \"43c7ca36-3d4d-496f-a3e8-67ec2e9b2243\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"
Oct 31 13:56:15.624909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.617425    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ngp\" (UniqueName: \"kubernetes.io/projected/43c7ca36-3d4d-496f-a3e8-67ec2e9b2243-kube-api-access-v2ngp\") pod \"payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx\" (UID: \"43c7ca36-3d4d-496f-a3e8-67ec2e9b2243\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"
Oct 31 13:56:15.624909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.617508    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c7ca36-3d4d-496f-a3e8-67ec2e9b2243-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx\" (UID: \"43c7ca36-3d4d-496f-a3e8-67ec2e9b2243\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"
Oct 31 13:56:15.624909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.618840    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c7ca36-3d4d-496f-a3e8-67ec2e9b2243-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx\" (UID: \"43c7ca36-3d4d-496f-a3e8-67ec2e9b2243\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"
Oct 31 13:56:15.638895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.637438    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:15.684788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.684707    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2ngp\" (UniqueName: \"kubernetes.io/projected/43c7ca36-3d4d-496f-a3e8-67ec2e9b2243-kube-api-access-v2ngp\") pod \"payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx\" (UID: \"43c7ca36-3d4d-496f-a3e8-67ec2e9b2243\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"
Oct 31 13:56:15.754892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:15.754450    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"
Oct 31 13:56:16.054031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:16.053994    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:16.148267 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:56:16.148235    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c7ca36_3d4d_496f_a3e8_67ec2e9b2243.slice/crio-301d976ea6ea476295ae2cab33d0a90730fa320870f06e5e44827438b0508e37 WatchSource:0}: Error finding container 301d976ea6ea476295ae2cab33d0a90730fa320870f06e5e44827438b0508e37: Status 404 returned error can't find the container with id 301d976ea6ea476295ae2cab33d0a90730fa320870f06e5e44827438b0508e37
Oct 31 13:56:16.148492 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:16.148368    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"]
Oct 31 13:56:16.241889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:16.238699    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:16.423405 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:16.423299    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:16.479839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:16.479790    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx" event={"ID":"43c7ca36-3d4d-496f-a3e8-67ec2e9b2243","Type":"ContainerStarted","Data":"a7b103dc57a8aa3b61c1a5ffdc2af39de34e83c0bba59f50615890056d2b1e78"}
Oct 31 13:56:16.480095 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:16.479903    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx" event={"ID":"43c7ca36-3d4d-496f-a3e8-67ec2e9b2243","Type":"ContainerStarted","Data":"301d976ea6ea476295ae2cab33d0a90730fa320870f06e5e44827438b0508e37"}
Oct 31 13:56:16.480095 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:16.480073    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"
Oct 31 13:56:16.503906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:16.503828    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx" podStartSLOduration=1.503812261 podStartE2EDuration="1.503812261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:56:16.499282066 +0000 UTC m=+3173.897630691" watchObservedRunningTime="2026-10-31 13:56:16.503812261 +0000 UTC m=+3173.902160884"
Oct 31 13:56:16.601023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:16.600988    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"]
Oct 31 13:56:16.601525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:16.601492    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46" podUID="eeed598b-fd5d-49a7-9f9b-46b88c29f64a" containerName="istio-proxy" containerID="cri-o://a23403d02a88799388f18a586e51dc81b94ed630903c5bb0547a017045ee918b" gracePeriod=30
Oct 31 13:56:16.611780 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:16.611741    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"]
Oct 31 13:56:16.612038 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:16.612007    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh" podUID="db9fd1b4-227c-46b9-973a-5a96f80597d4" containerName="istio-proxy" containerID="cri-o://2f32233c3ba5018ad4809b167420d7b543f134e71bb39dfab01fbe6b39a4cf5f" gracePeriod=30
Oct 31 13:56:16.832273 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:16.832188    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:17.529892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:17.524572    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:18.499632 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:18.499142    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:18.744099 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:56:18.744043    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:56:19.380134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:19.380095    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:20.019333 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:20.019298    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:20.174104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:20.174068    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:20.984894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:20.975033    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:21.458717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:21.458636    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:56:21.458717 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:21.458690    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46" podUID="eeed598b-fd5d-49a7-9f9b-46b88c29f64a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:56:21.883278 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:21.881855    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:22.017078 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.017047    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:56:22.021072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.021046    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:56:22.126948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.126908    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-credential-socket\") pod \"db9fd1b4-227c-46b9-973a-5a96f80597d4\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") "
Oct 31 13:56:22.127144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.126990    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-data\") pod \"db9fd1b4-227c-46b9-973a-5a96f80597d4\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") "
Oct 31 13:56:22.127144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127028    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db9fd1b4-227c-46b9-973a-5a96f80597d4-kube-api-access-s2nhw\" (UniqueName: \"kubernetes.io/projected/db9fd1b4-227c-46b9-973a-5a96f80597d4-kube-api-access-s2nhw\") pod \"db9fd1b4-227c-46b9-973a-5a96f80597d4\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") "
Oct 31 13:56:22.127144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127062    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db9fd1b4-227c-46b9-973a-5a96f80597d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9fd1b4-227c-46b9-973a-5a96f80597d4-istiod-ca-cert\") pod \"db9fd1b4-227c-46b9-973a-5a96f80597d4\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") "
Oct 31 13:56:22.127144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127097    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-credential-socket" pod "db9fd1b4-227c-46b9-973a-5a96f80597d4" (UID: "db9fd1b4-227c-46b9-973a-5a96f80597d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:22.127144 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127107    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-workload-socket\") pod \"db9fd1b4-227c-46b9-973a-5a96f80597d4\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") "
Oct 31 13:56:22.127442 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127153    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-token\" (UniqueName: \"kubernetes.io/projected/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-token\") pod \"db9fd1b4-227c-46b9-973a-5a96f80597d4\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") "
Oct 31 13:56:22.127442 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127205    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-credential-socket\") pod \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") "
Oct 31 13:56:22.127442 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127269    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-workload-certs\") pod \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") "
Oct 31 13:56:22.127442 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127283    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-data" pod "db9fd1b4-227c-46b9-973a-5a96f80597d4" (UID: "db9fd1b4-227c-46b9-973a-5a96f80597d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:22.127442 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127299    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-kube-api-access-l8j6t\" (UniqueName: \"kubernetes.io/projected/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-kube-api-access-l8j6t\") pod \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") "
Oct 31 13:56:22.127442 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127337    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-data\") pod \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") "
Oct 31 13:56:22.127442 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127431    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-podinfo\") pod \"db9fd1b4-227c-46b9-973a-5a96f80597d4\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") "
Oct 31 13:56:22.128029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127478    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-credential-socket" pod "eeed598b-fd5d-49a7-9f9b-46b88c29f64a" (UID: "eeed598b-fd5d-49a7-9f9b-46b88c29f64a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:22.128029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127489    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-envoy\") pod \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") "
Oct 31 13:56:22.128029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127536    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-workload-certs" pod "eeed598b-fd5d-49a7-9f9b-46b88c29f64a" (UID: "eeed598b-fd5d-49a7-9f9b-46b88c29f64a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:22.128029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127632    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-envoy\") pod \"db9fd1b4-227c-46b9-973a-5a96f80597d4\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") "
Oct 31 13:56:22.128029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127722    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-token\" (UniqueName: \"kubernetes.io/projected/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-token\") pod \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") "
Oct 31 13:56:22.128029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127633    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-data" pod "eeed598b-fd5d-49a7-9f9b-46b88c29f64a" (UID: "eeed598b-fd5d-49a7-9f9b-46b88c29f64a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:22.128029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127758    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9fd1b4-227c-46b9-973a-5a96f80597d4-istiod-ca-cert" pod "db9fd1b4-227c-46b9-973a-5a96f80597d4" (UID: "db9fd1b4-227c-46b9-973a-5a96f80597d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:56:22.128029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127804    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-workload-socket\") pod \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") "
Oct 31 13:56:22.128029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127839    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istiod-ca-cert\") pod \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") "
Oct 31 13:56:22.128029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127916    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-workload-certs\") pod \"db9fd1b4-227c-46b9-973a-5a96f80597d4\" (UID: \"db9fd1b4-227c-46b9-973a-5a96f80597d4\") "
Oct 31 13:56:22.128029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.127962    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-podinfo\") pod \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\" (UID: \"eeed598b-fd5d-49a7-9f9b-46b88c29f64a\") "
Oct 31 13:56:22.128665 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.128640    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-workload-certs" pod "db9fd1b4-227c-46b9-973a-5a96f80597d4" (UID: "db9fd1b4-227c-46b9-973a-5a96f80597d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:22.128730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.128661    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-workload-socket" pod "db9fd1b4-227c-46b9-973a-5a96f80597d4" (UID: "db9fd1b4-227c-46b9-973a-5a96f80597d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:22.128730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.128683    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istiod-ca-cert" pod "eeed598b-fd5d-49a7-9f9b-46b88c29f64a" (UID: "eeed598b-fd5d-49a7-9f9b-46b88c29f64a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:56:22.128892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.128849    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-workload-socket" pod "eeed598b-fd5d-49a7-9f9b-46b88c29f64a" (UID: "eeed598b-fd5d-49a7-9f9b-46b88c29f64a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:22.129418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.129390    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.129546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.129427    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.129546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.129443    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.129546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.129458    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.129546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.129474    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.129546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.129488    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9fd1b4-227c-46b9-973a-5a96f80597d4-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.129546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.129503    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.129546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.129517    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.129546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.129533    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.129546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.129547    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.130401 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.130343    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-envoy" pod "eeed598b-fd5d-49a7-9f9b-46b88c29f64a" (UID: "eeed598b-fd5d-49a7-9f9b-46b88c29f64a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:22.130568 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.130532    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-kube-api-access-l8j6t" pod "eeed598b-fd5d-49a7-9f9b-46b88c29f64a" (UID: "eeed598b-fd5d-49a7-9f9b-46b88c29f64a"). InnerVolumeSpecName "kube-api-access-l8j6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:56:22.131161 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.131133    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-podinfo" pod "db9fd1b4-227c-46b9-973a-5a96f80597d4" (UID: "db9fd1b4-227c-46b9-973a-5a96f80597d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:56:22.131289 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.131234    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-podinfo" pod "eeed598b-fd5d-49a7-9f9b-46b88c29f64a" (UID: "eeed598b-fd5d-49a7-9f9b-46b88c29f64a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:56:22.132028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.132005    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9fd1b4-227c-46b9-973a-5a96f80597d4-kube-api-access-s2nhw" pod "db9fd1b4-227c-46b9-973a-5a96f80597d4" (UID: "db9fd1b4-227c-46b9-973a-5a96f80597d4"). InnerVolumeSpecName "kube-api-access-s2nhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:56:22.132193 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.132167    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-token" pod "eeed598b-fd5d-49a7-9f9b-46b88c29f64a" (UID: "eeed598b-fd5d-49a7-9f9b-46b88c29f64a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:56:22.132251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.132199    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-envoy" pod "db9fd1b4-227c-46b9-973a-5a96f80597d4" (UID: "db9fd1b4-227c-46b9-973a-5a96f80597d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:56:22.133283 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.133261    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-token" pod "db9fd1b4-227c-46b9-973a-5a96f80597d4" (UID: "db9fd1b4-227c-46b9-973a-5a96f80597d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:56:22.233265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.233128    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8j6t\" (UniqueName: \"kubernetes.io/projected/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-kube-api-access-l8j6t\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.233265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.233175    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.233265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.233195    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.233265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.233210    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.233265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.233226    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.233265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.233241    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeed598b-fd5d-49a7-9f9b-46b88c29f64a-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.233265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.233257    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2nhw\" (UniqueName: \"kubernetes.io/projected/db9fd1b4-227c-46b9-973a-5a96f80597d4-kube-api-access-s2nhw\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.233265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.233270    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9fd1b4-227c-46b9-973a-5a96f80597d4-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:22.586157 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.586055    4428 generic.go:356] "Generic (PLEG): container finished" podID="db9fd1b4-227c-46b9-973a-5a96f80597d4" containerID="2f32233c3ba5018ad4809b167420d7b543f134e71bb39dfab01fbe6b39a4cf5f" exitCode=0
Oct 31 13:56:22.586157 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.586101    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"
Oct 31 13:56:22.586157 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.586145    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh" event={"ID":"db9fd1b4-227c-46b9-973a-5a96f80597d4","Type":"ContainerDied","Data":"2f32233c3ba5018ad4809b167420d7b543f134e71bb39dfab01fbe6b39a4cf5f"}
Oct 31 13:56:22.586754 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.586215    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh" event={"ID":"db9fd1b4-227c-46b9-973a-5a96f80597d4","Type":"ContainerDied","Data":"e375d5aa084f15b61a4f3aa565d56df5c68a88a2b29762fd60c34ded87c49f98"}
Oct 31 13:56:22.586754 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.586248    4428 scope.go:122] "RemoveContainer" containerID="2f32233c3ba5018ad4809b167420d7b543f134e71bb39dfab01fbe6b39a4cf5f"
Oct 31 13:56:22.587710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.587688    4428 generic.go:356] "Generic (PLEG): container finished" podID="eeed598b-fd5d-49a7-9f9b-46b88c29f64a" containerID="a23403d02a88799388f18a586e51dc81b94ed630903c5bb0547a017045ee918b" exitCode=0
Oct 31 13:56:22.587830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.587737    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"
Oct 31 13:56:22.587830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.587769    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46" event={"ID":"eeed598b-fd5d-49a7-9f9b-46b88c29f64a","Type":"ContainerDied","Data":"a23403d02a88799388f18a586e51dc81b94ed630903c5bb0547a017045ee918b"}
Oct 31 13:56:22.587830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.587810    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46" event={"ID":"eeed598b-fd5d-49a7-9f9b-46b88c29f64a","Type":"ContainerDied","Data":"2abd4703e3b716826b745226d525bc0b39fb1898159b618e8a487c1cfa353a99"}
Oct 31 13:56:22.599999 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.599977    4428 scope.go:122] "RemoveContainer" containerID="2f32233c3ba5018ad4809b167420d7b543f134e71bb39dfab01fbe6b39a4cf5f"
Oct 31 13:56:22.600319 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:56:22.600298    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f32233c3ba5018ad4809b167420d7b543f134e71bb39dfab01fbe6b39a4cf5f\": container with ID starting with 2f32233c3ba5018ad4809b167420d7b543f134e71bb39dfab01fbe6b39a4cf5f not found: ID does not exist" containerID="2f32233c3ba5018ad4809b167420d7b543f134e71bb39dfab01fbe6b39a4cf5f"
Oct 31 13:56:22.600394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.600328    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f32233c3ba5018ad4809b167420d7b543f134e71bb39dfab01fbe6b39a4cf5f"} err="failed to get container status \"2f32233c3ba5018ad4809b167420d7b543f134e71bb39dfab01fbe6b39a4cf5f\": rpc error: code = NotFound desc = could not find container \"2f32233c3ba5018ad4809b167420d7b543f134e71bb39dfab01fbe6b39a4cf5f\": container with ID starting with 2f32233c3ba5018ad4809b167420d7b543f134e71bb39dfab01fbe6b39a4cf5f not found: ID does not exist"
Oct 31 13:56:22.600394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.600351    4428 scope.go:122] "RemoveContainer" containerID="a23403d02a88799388f18a586e51dc81b94ed630903c5bb0547a017045ee918b"
Oct 31 13:56:22.611833 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.611813    4428 scope.go:122] "RemoveContainer" containerID="a23403d02a88799388f18a586e51dc81b94ed630903c5bb0547a017045ee918b"
Oct 31 13:56:22.612198 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:56:22.612177    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a23403d02a88799388f18a586e51dc81b94ed630903c5bb0547a017045ee918b\": container with ID starting with a23403d02a88799388f18a586e51dc81b94ed630903c5bb0547a017045ee918b not found: ID does not exist" containerID="a23403d02a88799388f18a586e51dc81b94ed630903c5bb0547a017045ee918b"
Oct 31 13:56:22.612252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.612206    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23403d02a88799388f18a586e51dc81b94ed630903c5bb0547a017045ee918b"} err="failed to get container status \"a23403d02a88799388f18a586e51dc81b94ed630903c5bb0547a017045ee918b\": rpc error: code = NotFound desc = could not find container \"a23403d02a88799388f18a586e51dc81b94ed630903c5bb0547a017045ee918b\": container with ID starting with a23403d02a88799388f18a586e51dc81b94ed630903c5bb0547a017045ee918b not found: ID does not exist"
Oct 31 13:56:22.619993 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.619967    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"]
Oct 31 13:56:22.624525 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.624502    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-w6d46"]
Oct 31 13:56:22.648463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.646096    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"]
Oct 31 13:56:22.656151 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.656122    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh"]
Oct 31 13:56:22.761960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.761926    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db9fd1b4-227c-46b9-973a-5a96f80597d4" path="/var/lib/kubelet/pods/db9fd1b4-227c-46b9-973a-5a96f80597d4/volumes"
Oct 31 13:56:22.762453 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.762437    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeed598b-fd5d-49a7-9f9b-46b88c29f64a" path="/var/lib/kubelet/pods/eeed598b-fd5d-49a7-9f9b-46b88c29f64a/volumes"
Oct 31 13:56:22.854929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.854833    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:56:22.854929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.854917    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-7wkjh" podUID="db9fd1b4-227c-46b9-973a-5a96f80597d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:56:22.958786 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:22.954621    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:24.203918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:24.200794    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:25.261860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:25.261436    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:26.176679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:26.176642    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:26.760766 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:26.760715    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:26.866424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:26.866388    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"]
Oct 31 13:56:26.867250 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:26.867233    4428 cpu_manager.go:414] "containerMap: removing container" podUID="eeed598b-fd5d-49a7-9f9b-46b88c29f64a" containerName="istio-proxy"
Oct 31 13:56:26.867318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:26.867251    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeed598b-fd5d-49a7-9f9b-46b88c29f64a" containerName="istio-proxy"
Oct 31 13:56:26.867318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:26.867290    4428 cpu_manager.go:414] "containerMap: removing container" podUID="db9fd1b4-227c-46b9-973a-5a96f80597d4" containerName="istio-proxy"
Oct 31 13:56:26.867318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:26.867295    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db9fd1b4-227c-46b9-973a-5a96f80597d4" containerName="istio-proxy"
Oct 31 13:56:26.867430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:26.867420    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeed598b-fd5d-49a7-9f9b-46b88c29f64a" containerName="istio-proxy"
Oct 31 13:56:26.867473 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:26.867434    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="db9fd1b4-227c-46b9-973a-5a96f80597d4" containerName="istio-proxy"
Oct 31 13:56:26.870839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:26.870818    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"
Oct 31 13:56:26.879908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:26.879780    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-403dff\""
Oct 31 13:56:26.880075 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:26.879951    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-403dff-dockercfg-79lw7\""
Oct 31 13:56:26.884532 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:26.884507    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"]
Oct 31 13:56:26.925130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:26.925093    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:27.148944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.055065    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mkjr\" (UniqueName: \"kubernetes.io/projected/9825f1d9-236b-4516-94b1-058e389a702c-kube-api-access-4mkjr\") pod \"payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg\" (UID: \"9825f1d9-236b-4516-94b1-058e389a702c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"
Oct 31 13:56:27.148944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.055214    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9825f1d9-236b-4516-94b1-058e389a702c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg\" (UID: \"9825f1d9-236b-4516-94b1-058e389a702c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"
Oct 31 13:56:27.148944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.058338    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"]
Oct 31 13:56:27.152296 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.152266    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"]
Oct 31 13:56:27.160117 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.160083    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"
Oct 31 13:56:27.219530 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.219474    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bss4h\" (UniqueName: \"kubernetes.io/projected/66b6b2a9-e6b8-4848-9fb7-4886a7db34e4-kube-api-access-bss4h\") pod \"payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b\" (UID: \"66b6b2a9-e6b8-4848-9fb7-4886a7db34e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"
Oct 31 13:56:27.219954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.219928    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mkjr\" (UniqueName: \"kubernetes.io/projected/9825f1d9-236b-4516-94b1-058e389a702c-kube-api-access-4mkjr\") pod \"payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg\" (UID: \"9825f1d9-236b-4516-94b1-058e389a702c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"
Oct 31 13:56:27.220191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.220173    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b6b2a9-e6b8-4848-9fb7-4886a7db34e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b\" (UID: \"66b6b2a9-e6b8-4848-9fb7-4886a7db34e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"
Oct 31 13:56:27.220343 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.220329    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9825f1d9-236b-4516-94b1-058e389a702c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg\" (UID: \"9825f1d9-236b-4516-94b1-058e389a702c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"
Oct 31 13:56:27.221812 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.221785    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9825f1d9-236b-4516-94b1-058e389a702c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg\" (UID: \"9825f1d9-236b-4516-94b1-058e389a702c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"
Oct 31 13:56:27.251101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.251060    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mkjr\" (UniqueName: \"kubernetes.io/projected/9825f1d9-236b-4516-94b1-058e389a702c-kube-api-access-4mkjr\") pod \"payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg\" (UID: \"9825f1d9-236b-4516-94b1-058e389a702c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"
Oct 31 13:56:27.322038 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.321994    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bss4h\" (UniqueName: \"kubernetes.io/projected/66b6b2a9-e6b8-4848-9fb7-4886a7db34e4-kube-api-access-bss4h\") pod \"payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b\" (UID: \"66b6b2a9-e6b8-4848-9fb7-4886a7db34e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"
Oct 31 13:56:27.322238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.322154    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b6b2a9-e6b8-4848-9fb7-4886a7db34e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b\" (UID: \"66b6b2a9-e6b8-4848-9fb7-4886a7db34e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"
Oct 31 13:56:27.322934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.322912    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b6b2a9-e6b8-4848-9fb7-4886a7db34e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b\" (UID: \"66b6b2a9-e6b8-4848-9fb7-4886a7db34e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"
Oct 31 13:56:27.336151 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.336116    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bss4h\" (UniqueName: \"kubernetes.io/projected/66b6b2a9-e6b8-4848-9fb7-4886a7db34e4-kube-api-access-bss4h\") pod \"payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b\" (UID: \"66b6b2a9-e6b8-4848-9fb7-4886a7db34e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"
Oct 31 13:56:27.491438 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.491393    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"
Oct 31 13:56:27.534575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.533346    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"
Oct 31 13:56:27.559574 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.559537    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"
Oct 31 13:56:27.730349 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.730312    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:27.909275 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:27.904979    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:28.037984 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:28.037946    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"]
Oct 31 13:56:28.043918 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:28.043509    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"]
Oct 31 13:56:28.670398 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:28.670351    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg" event={"ID":"9825f1d9-236b-4516-94b1-058e389a702c","Type":"ContainerStarted","Data":"c1c31dcebf4653ad04b4e1f27a6043a7063986a342db694da98fd6da1112c638"}
Oct 31 13:56:28.672304 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:28.672274    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b" event={"ID":"66b6b2a9-e6b8-4848-9fb7-4886a7db34e4","Type":"ContainerStarted","Data":"c05fa7d2d8f7f587b7c3014b7905e918311c21cafd05aa2d164f2f88f8bc015f"}
Oct 31 13:56:29.012859 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:29.012713    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:29.683658 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:29.683616    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b" event={"ID":"66b6b2a9-e6b8-4848-9fb7-4886a7db34e4","Type":"ContainerStarted","Data":"efbf06733dc72043c1bb4e624496746c203f071856e093c08cec16fb2ad5240d"}
Oct 31 13:56:29.683860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:29.683710    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"
Oct 31 13:56:29.712940 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:29.712099    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b" podStartSLOduration=2.255567445 podStartE2EDuration="3.712074729s" totalImagesPullingTime="1.456507284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:56:29.703529758 +0000 UTC m=+3187.101878377" watchObservedRunningTime="2026-10-31 13:56:29.712074729 +0000 UTC m=+3187.110423351"
Oct 31 13:56:29.783883 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:29.783829    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:30.615893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:30.609826    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:30.694331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:30.694267    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg" event={"ID":"9825f1d9-236b-4516-94b1-058e389a702c","Type":"ContainerStarted","Data":"9b6082754709088e13b8f7e33021f896bdfc3e0cee584aa229c298e9e0d9959b"}
Oct 31 13:56:30.694557 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:30.694537    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"
Oct 31 13:56:30.717049 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:30.716990    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg" podStartSLOduration=3.14612146 podStartE2EDuration="4.716971864s" totalImagesPullingTime="1.570850404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:56:30.71403019 +0000 UTC m=+3188.112378837" watchObservedRunningTime="2026-10-31 13:56:30.716971864 +0000 UTC m=+3188.115320488"
Oct 31 13:56:31.397922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.393753    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:31.584247 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.584209    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"]
Oct 31 13:56:31.595423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.590350    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.601207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.601173    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"]
Oct 31 13:56:31.601547 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.601523    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-14ba5a\"/\"rate-test-model-14ba5a-kserve-self-signed-certs\""
Oct 31 13:56:31.601699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.601556    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-14ba5a\"/\"openshift-service-ca.crt\""
Oct 31 13:56:31.601699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.601624    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-14ba5a\"/\"default-dockercfg-ztxfb\""
Oct 31 13:56:31.601923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.601859    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-14ba5a\"/\"kube-root-ca.crt\""
Oct 31 13:56:31.669625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.669544    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:31.686196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.686158    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-home\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.686372 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.686207    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-tls-certs\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.686414 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.686381    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-tmp-dir\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.686529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.686507    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz92s\" (UniqueName: \"kubernetes.io/projected/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-kube-api-access-vz92s\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.686612 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.686598    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-model-cache\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.686679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.686660    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-dshm\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.804910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.800380    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-home\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.804910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.800447    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-tls-certs\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.804910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.800594    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-tmp-dir\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.804910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.800704    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz92s\" (UniqueName: \"kubernetes.io/projected/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-kube-api-access-vz92s\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.804910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.800812    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-model-cache\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.804910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.800839    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-dshm\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.804910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.802431    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-tmp-dir\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.804910 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.802683    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-model-cache\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.823897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.822794    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-tls-certs\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.831896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.829631    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-home\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.840889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.835134    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-dshm\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.840889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.839747    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz92s\" (UniqueName: \"kubernetes.io/projected/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-kube-api-access-vz92s\") pod \"rate-test-model-14ba5a-kserve-66d59c65d9-w279h\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:31.918538 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:31.918140    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:32.232782 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:32.232748    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"]
Oct 31 13:56:32.233213 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:56:32.233187    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6bd76b8_0062_40ce_a682_5ad4ee1c0ccb.slice/crio-0bfa151a766353cb5cf5cdb92e92f88ad42ce16b5fe38be110b9643dd13cf28e WatchSource:0}: Error finding container 0bfa151a766353cb5cf5cdb92e92f88ad42ce16b5fe38be110b9643dd13cf28e: Status 404 returned error can't find the container with id 0bfa151a766353cb5cf5cdb92e92f88ad42ce16b5fe38be110b9643dd13cf28e
Oct 31 13:56:32.306000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:32.296574    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:32.711663 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:32.711622    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h" event={"ID":"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb","Type":"ContainerStarted","Data":"65b31d3f2b141f856e6b740f24093e876378d9d2d88f522b396cb52651a1a506"}
Oct 31 13:56:32.712107 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:32.711673    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h" event={"ID":"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb","Type":"ContainerStarted","Data":"0bfa151a766353cb5cf5cdb92e92f88ad42ce16b5fe38be110b9643dd13cf28e"}
Oct 31 13:56:32.743524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:32.743455    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h" podStartSLOduration=1.743434989 podStartE2EDuration="1.743434989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:56:32.739095552 +0000 UTC m=+3190.137444181" watchObservedRunningTime="2026-10-31 13:56:32.743434989 +0000 UTC m=+3190.141783616"
Oct 31 13:56:32.918926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:32.918823    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:33.416854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:33.397299    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:33.568594 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:33.568563    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:34.074236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:34.074159    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:34.236479 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:34.236406    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:34.394994 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:34.394894    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:34.593007 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:34.592970    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:35.026360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:35.026322    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:35.607534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:35.605723    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:36.587904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:36.587846    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:37.402029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:37.401997    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:37.520130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:37.520104    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w9wjk_e328741d-31fd-40db-8cbb-8c9b518f4cde/payload-processing/0.log"
Oct 31 13:56:37.700272 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:37.700202    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w9wjk_e328741d-31fd-40db-8cbb-8c9b518f4cde/payload-processing/0.log"
Oct 31 13:56:38.226451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:38.226414    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:39.213955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:39.210853    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:39.977407 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:39.977369    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"]
Oct 31 13:56:39.986392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:39.986124    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:56:39.993193 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:39.993162    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e50b22-serving-cert\""
Oct 31 13:56:40.001334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.001288    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"]
Oct 31 13:56:40.076772 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.076740    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:40.089224 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.089189    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82f12bdb-2c78-4081-9cfc-653a252ab25b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj\" (UID: \"82f12bdb-2c78-4081-9cfc-653a252ab25b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:56:40.089854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.089780    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82f12bdb-2c78-4081-9cfc-653a252ab25b-metrics-certs\") pod \"maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj\" (UID: \"82f12bdb-2c78-4081-9cfc-653a252ab25b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:56:40.090098 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.090079    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f12bdb-2c78-4081-9cfc-653a252ab25b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj\" (UID: \"82f12bdb-2c78-4081-9cfc-653a252ab25b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:56:40.090321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.090303    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzmkv\" (UniqueName: \"kubernetes.io/projected/82f12bdb-2c78-4081-9cfc-653a252ab25b-kube-api-access-bzmkv\") pod \"maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj\" (UID: \"82f12bdb-2c78-4081-9cfc-653a252ab25b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:56:40.230961 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.230830    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:40.241394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.241369    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82f12bdb-2c78-4081-9cfc-653a252ab25b-metrics-certs\") pod \"maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj\" (UID: \"82f12bdb-2c78-4081-9cfc-653a252ab25b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:56:40.241545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.241417    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f12bdb-2c78-4081-9cfc-653a252ab25b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj\" (UID: \"82f12bdb-2c78-4081-9cfc-653a252ab25b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:56:40.241545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.241469    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzmkv\" (UniqueName: \"kubernetes.io/projected/82f12bdb-2c78-4081-9cfc-653a252ab25b-kube-api-access-bzmkv\") pod \"maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj\" (UID: \"82f12bdb-2c78-4081-9cfc-653a252ab25b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:56:40.241662 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.241640    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82f12bdb-2c78-4081-9cfc-653a252ab25b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj\" (UID: \"82f12bdb-2c78-4081-9cfc-653a252ab25b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:56:40.242178 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.242156    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f12bdb-2c78-4081-9cfc-653a252ab25b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj\" (UID: \"82f12bdb-2c78-4081-9cfc-653a252ab25b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:56:40.244227 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.244202    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82f12bdb-2c78-4081-9cfc-653a252ab25b-metrics-certs\") pod \"maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj\" (UID: \"82f12bdb-2c78-4081-9cfc-653a252ab25b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:56:40.244322 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.244248    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82f12bdb-2c78-4081-9cfc-653a252ab25b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj\" (UID: \"82f12bdb-2c78-4081-9cfc-653a252ab25b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:56:40.254402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.254376    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzmkv\" (UniqueName: \"kubernetes.io/projected/82f12bdb-2c78-4081-9cfc-653a252ab25b-kube-api-access-bzmkv\") pod \"maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj\" (UID: \"82f12bdb-2c78-4081-9cfc-653a252ab25b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:56:40.318955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.318915    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:56:40.383858 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.383829    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:40.609590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.609560    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"]
Oct 31 13:56:40.610017 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:56:40.609988    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f12bdb_2c78_4081_9cfc_653a252ab25b.slice/crio-f8e6ab7ed6bf9e2ea9f3f15c0653d43ad87a59fcb3de80fd88b47f0bc337225e WatchSource:0}: Error finding container f8e6ab7ed6bf9e2ea9f3f15c0653d43ad87a59fcb3de80fd88b47f0bc337225e: Status 404 returned error can't find the container with id f8e6ab7ed6bf9e2ea9f3f15c0653d43ad87a59fcb3de80fd88b47f0bc337225e
Oct 31 13:56:40.745081 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.745044    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"
Oct 31 13:56:40.778699 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:40.778666    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj" event={"ID":"82f12bdb-2c78-4081-9cfc-653a252ab25b","Type":"ContainerStarted","Data":"f8e6ab7ed6bf9e2ea9f3f15c0653d43ad87a59fcb3de80fd88b47f0bc337225e"}
Oct 31 13:56:41.283611 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:41.283580    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:41.704093 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:41.704051    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"
Oct 31 13:56:41.755572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:41.755530    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:41.789956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:41.789917    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj" event={"ID":"82f12bdb-2c78-4081-9cfc-653a252ab25b","Type":"ContainerStarted","Data":"981bde87e95607822b7e264bf4a92c36264f924b7715457574173e03c098227a"}
Oct 31 13:56:41.790176 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:41.790076    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:56:41.816306 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:41.816246    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj" podStartSLOduration=2.221507352 podStartE2EDuration="2.816226602s" totalImagesPullingTime="594.71925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:56:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:56:41.813393392 +0000 UTC m=+3199.211742028" watchObservedRunningTime="2026-10-31 13:56:41.816226602 +0000 UTC m=+3199.214575228"
Oct 31 13:56:41.917280 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:41.917245    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:41.919170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:41.919150    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:41.937906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:41.937838    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:42.151893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:42.122469    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:42.372120 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:42.372082    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:42.809528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:42.809488    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 13:56:43.594422 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:43.594383    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:44.407486 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:44.407444    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:45.124483 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:45.124444    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:46.038435 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:46.038405    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:46.980050 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:46.979955    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:47.092714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.092679    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"]
Oct 31 13:56:47.099494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.099462    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.109607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.109566    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a81979\"/\"ipp-route-a81979-kserve-self-signed-certs\""
Oct 31 13:56:47.109847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.109684    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a81979\"/\"default-dockercfg-spzcq\""
Oct 31 13:56:47.109847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.109732    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a81979\"/\"kube-root-ca.crt\""
Oct 31 13:56:47.109847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.109792    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a81979\"/\"openshift-service-ca.crt\""
Oct 31 13:56:47.115370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.115339    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"]
Oct 31 13:56:47.202526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.202488    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:47.266809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.266721    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc9xw\" (UniqueName: \"kubernetes.io/projected/be1f73c9-7c0e-4972-8cae-d8a029d311f3-kube-api-access-vc9xw\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.266809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.266771    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-dshm\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.267095 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.267027    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-home\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.267180 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.267154    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-model-cache\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.267263 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.267247    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-tmp-dir\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.267354 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.267329    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be1f73c9-7c0e-4972-8cae-d8a029d311f3-tls-certs\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.391958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.391369    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc9xw\" (UniqueName: \"kubernetes.io/projected/be1f73c9-7c0e-4972-8cae-d8a029d311f3-kube-api-access-vc9xw\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.391958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.391431    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-dshm\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.391958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.391518    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-home\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.391958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.391598    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-model-cache\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.391958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.391661    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-tmp-dir\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.391958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.391713    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be1f73c9-7c0e-4972-8cae-d8a029d311f3-tls-certs\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.395895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.395357    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-home\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.400053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.396571    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-model-cache\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.400053 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.397217    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-tmp-dir\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.409644 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.409603    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be1f73c9-7c0e-4972-8cae-d8a029d311f3-tls-certs\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.452991 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.452919    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc9xw\" (UniqueName: \"kubernetes.io/projected/be1f73c9-7c0e-4972-8cae-d8a029d311f3-kube-api-access-vc9xw\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.453504 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.453439    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-dshm\") pod \"ipp-route-a81979-kserve-7bd7bcc49-hsd28\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") " pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.725497 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.725462    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:47.819914 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:47.819125    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:56:48.033590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:48.030857    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:48.066855 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:48.066829    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"]
Oct 31 13:56:48.067195 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:56:48.067170    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe1f73c9_7c0e_4972_8cae_d8a029d311f3.slice/crio-719ce59196d306d4f21c8d51f8bb0b94c92a2edf08f7108462d2c7d611c0e184 WatchSource:0}: Error finding container 719ce59196d306d4f21c8d51f8bb0b94c92a2edf08f7108462d2c7d611c0e184: Status 404 returned error can't find the container with id 719ce59196d306d4f21c8d51f8bb0b94c92a2edf08f7108462d2c7d611c0e184
Oct 31 13:56:48.742115 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:56:48.742078    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:56:48.857061 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:48.848215    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:48.880549 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:48.877331    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28" event={"ID":"be1f73c9-7c0e-4972-8cae-d8a029d311f3","Type":"ContainerStarted","Data":"5a1cba1862752cc95f37a4ec4f0676a6578348d92ec4d09919a708855dcafd88"}
Oct 31 13:56:48.880549 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:48.877404    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28" event={"ID":"be1f73c9-7c0e-4972-8cae-d8a029d311f3","Type":"ContainerStarted","Data":"719ce59196d306d4f21c8d51f8bb0b94c92a2edf08f7108462d2c7d611c0e184"}
Oct 31 13:56:48.905051 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:48.904992    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28" podStartSLOduration=1.904979019 podStartE2EDuration="1.904979019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:56:48.901161643 +0000 UTC m=+3206.299510267" watchObservedRunningTime="2026-10-31 13:56:48.904979019 +0000 UTC m=+3206.303327643"
Oct 31 13:56:49.712896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:49.706143    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:49.729024 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:49.727982    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:50.443972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:50.443934    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:51.235737 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:51.235698    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:51.979758 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:51.979708    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"]
Oct 31 13:56:51.980082 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:51.980039    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf" podUID="2e6f1d83-780e-4779-a9b6-fe0b967f149a" containerName="maas-api" containerID="cri-o://9ca6c25f7a29c98f7a03c3ef196b598274c938cffb86c4fe062d099eabbe726e" gracePeriod=30
Oct 31 13:56:52.090455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.090374    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:52.248476 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.248368    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:52.414893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.411380    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:52.445523 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.444341    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:56:52.517524 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.517431    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e6f1d83-780e-4779-a9b6-fe0b967f149a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6f1d83-780e-4779-a9b6-fe0b967f149a-maas-api-tls\") pod \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\" (UID: \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\") "
Oct 31 13:56:52.517947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.517918    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e6f1d83-780e-4779-a9b6-fe0b967f149a-kube-api-access-gb2jv\" (UniqueName: \"kubernetes.io/projected/2e6f1d83-780e-4779-a9b6-fe0b967f149a-kube-api-access-gb2jv\") pod \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\" (UID: \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\") "
Oct 31 13:56:52.518111 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.518100    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e6f1d83-780e-4779-a9b6-fe0b967f149a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6f1d83-780e-4779-a9b6-fe0b967f149a-metrics-certs\") pod \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\" (UID: \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\") "
Oct 31 13:56:52.518340 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.518330    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e6f1d83-780e-4779-a9b6-fe0b967f149a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6f1d83-780e-4779-a9b6-fe0b967f149a-openshift-service-ca\") pod \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\" (UID: \"2e6f1d83-780e-4779-a9b6-fe0b967f149a\") "
Oct 31 13:56:52.520884 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.520836    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6f1d83-780e-4779-a9b6-fe0b967f149a-openshift-service-ca" pod "2e6f1d83-780e-4779-a9b6-fe0b967f149a" (UID: "2e6f1d83-780e-4779-a9b6-fe0b967f149a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:56:52.524168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.524127    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6f1d83-780e-4779-a9b6-fe0b967f149a-maas-api-tls" pod "2e6f1d83-780e-4779-a9b6-fe0b967f149a" (UID: "2e6f1d83-780e-4779-a9b6-fe0b967f149a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:56:52.524418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.524383    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6f1d83-780e-4779-a9b6-fe0b967f149a-metrics-certs" pod "2e6f1d83-780e-4779-a9b6-fe0b967f149a" (UID: "2e6f1d83-780e-4779-a9b6-fe0b967f149a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:56:52.532425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.532314    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6f1d83-780e-4779-a9b6-fe0b967f149a-kube-api-access-gb2jv" pod "2e6f1d83-780e-4779-a9b6-fe0b967f149a" (UID: "2e6f1d83-780e-4779-a9b6-fe0b967f149a"). InnerVolumeSpecName "kube-api-access-gb2jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:56:52.622511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.622476    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6f1d83-780e-4779-a9b6-fe0b967f149a-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:52.622511 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.622514    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb2jv\" (UniqueName: \"kubernetes.io/projected/2e6f1d83-780e-4779-a9b6-fe0b967f149a-kube-api-access-gb2jv\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:52.622742 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.622528    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6f1d83-780e-4779-a9b6-fe0b967f149a-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:52.622742 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.622537    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6f1d83-780e-4779-a9b6-fe0b967f149a-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:56:52.814009 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.813971    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:52.909604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.909563    4428 generic.go:356] "Generic (PLEG): container finished" podID="2e6f1d83-780e-4779-a9b6-fe0b967f149a" containerID="9ca6c25f7a29c98f7a03c3ef196b598274c938cffb86c4fe062d099eabbe726e" exitCode=0
Oct 31 13:56:52.909936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.909625    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"
Oct 31 13:56:52.909936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.909655    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf" event={"ID":"2e6f1d83-780e-4779-a9b6-fe0b967f149a","Type":"ContainerDied","Data":"9ca6c25f7a29c98f7a03c3ef196b598274c938cffb86c4fe062d099eabbe726e"}
Oct 31 13:56:52.909936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.909729    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf" event={"ID":"2e6f1d83-780e-4779-a9b6-fe0b967f149a","Type":"ContainerDied","Data":"7a657980fcd90c334ea90c724ab8af610b30a0e2a516ace7065739b41df93ec1"}
Oct 31 13:56:52.909936 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.909765    4428 scope.go:122] "RemoveContainer" containerID="9ca6c25f7a29c98f7a03c3ef196b598274c938cffb86c4fe062d099eabbe726e"
Oct 31 13:56:52.922806 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.922786    4428 scope.go:122] "RemoveContainer" containerID="9ca6c25f7a29c98f7a03c3ef196b598274c938cffb86c4fe062d099eabbe726e"
Oct 31 13:56:52.923165 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:56:52.923146    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca6c25f7a29c98f7a03c3ef196b598274c938cffb86c4fe062d099eabbe726e\": container with ID starting with 9ca6c25f7a29c98f7a03c3ef196b598274c938cffb86c4fe062d099eabbe726e not found: ID does not exist" containerID="9ca6c25f7a29c98f7a03c3ef196b598274c938cffb86c4fe062d099eabbe726e"
Oct 31 13:56:52.923229 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.923173    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca6c25f7a29c98f7a03c3ef196b598274c938cffb86c4fe062d099eabbe726e"} err="failed to get container status \"9ca6c25f7a29c98f7a03c3ef196b598274c938cffb86c4fe062d099eabbe726e\": rpc error: code = NotFound desc = could not find container \"9ca6c25f7a29c98f7a03c3ef196b598274c938cffb86c4fe062d099eabbe726e\": container with ID starting with 9ca6c25f7a29c98f7a03c3ef196b598274c938cffb86c4fe062d099eabbe726e not found: ID does not exist"
Oct 31 13:56:52.934036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.933986    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"]
Oct 31 13:56:52.938795 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.938770    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be675-64b5d67bf-xgjrf"]
Oct 31 13:56:52.975729 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:52.975693    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:53.289797 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:53.289553    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:54.001890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:54.001633    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:54.758209 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:54.758173    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e6f1d83-780e-4779-a9b6-fe0b967f149a" path="/var/lib/kubelet/pods/2e6f1d83-780e-4779-a9b6-fe0b967f149a/volumes"
Oct 31 13:56:54.801896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:54.799298    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:55.631890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:55.626842    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:56.520895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:56.517054    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:57.472166 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:57.472133    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:57.725741 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:57.725652    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:57.739508 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:57.739482    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:57.964663 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:57.964634    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 13:56:58.366960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:58.360347    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:56:59.994894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:56:59.994461    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:00.181999 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:00.180390    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:00.352828 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:00.352731    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:01.812891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:01.806560    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:02.817992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:02.817948    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:03.685647 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:03.685549    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:04.503126 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:04.503090    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:05.356907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:05.356815    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:06.463899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:06.456429    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:07.519095 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:07.519058    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:08.299891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:08.299780    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:09.125893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:09.124509    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:09.910819 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:09.910775    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:10.232392 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:10.232336    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"]
Oct 31 13:57:10.349913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:10.349862    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:10.538957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:10.537392    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:10.790032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:10.789860    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:11.076168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:11.076061    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:12.133897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:12.129130    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:12.915020 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:12.914533    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:13.734228 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:13.734190    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:14.631552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:14.631508    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:15.112171 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:15.112124    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:15.232795 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:15.232747    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx" podUID="43c7ca36-3d4d-496f-a3e8-67ec2e9b2243" containerName="payload-processing" containerID="cri-o://a7b103dc57a8aa3b61c1a5ffdc2af39de34e83c0bba59f50615890056d2b1e78" gracePeriod=25
Oct 31 13:57:15.399945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:15.396304    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:15.607080 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:15.607050    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"
Oct 31 13:57:15.667071 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:15.666975    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c7ca36-3d4d-496f-a3e8-67ec2e9b2243-kube-api-access-v2ngp\" (UniqueName: \"kubernetes.io/projected/43c7ca36-3d4d-496f-a3e8-67ec2e9b2243-kube-api-access-v2ngp\") pod \"43c7ca36-3d4d-496f-a3e8-67ec2e9b2243\" (UID: \"43c7ca36-3d4d-496f-a3e8-67ec2e9b2243\") "
Oct 31 13:57:15.667071 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:15.667044    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c7ca36-3d4d-496f-a3e8-67ec2e9b2243-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c7ca36-3d4d-496f-a3e8-67ec2e9b2243-plugins-config-volume\") pod \"43c7ca36-3d4d-496f-a3e8-67ec2e9b2243\" (UID: \"43c7ca36-3d4d-496f-a3e8-67ec2e9b2243\") "
Oct 31 13:57:15.667449 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:15.667422    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c7ca36-3d4d-496f-a3e8-67ec2e9b2243-plugins-config-volume" pod "43c7ca36-3d4d-496f-a3e8-67ec2e9b2243" (UID: "43c7ca36-3d4d-496f-a3e8-67ec2e9b2243"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:57:15.667839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:15.667822    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c7ca36-3d4d-496f-a3e8-67ec2e9b2243-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:15.669425 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:15.669398    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c7ca36-3d4d-496f-a3e8-67ec2e9b2243-kube-api-access-v2ngp" pod "43c7ca36-3d4d-496f-a3e8-67ec2e9b2243" (UID: "43c7ca36-3d4d-496f-a3e8-67ec2e9b2243"). InnerVolumeSpecName "kube-api-access-v2ngp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:57:15.769733 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:15.769688    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2ngp\" (UniqueName: \"kubernetes.io/projected/43c7ca36-3d4d-496f-a3e8-67ec2e9b2243-kube-api-access-v2ngp\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:16.132643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:16.132605    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:16.216301 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:16.216260    4428 generic.go:356] "Generic (PLEG): container finished" podID="43c7ca36-3d4d-496f-a3e8-67ec2e9b2243" containerID="a7b103dc57a8aa3b61c1a5ffdc2af39de34e83c0bba59f50615890056d2b1e78" exitCode=0
Oct 31 13:57:16.216536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:16.216314    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"
Oct 31 13:57:16.216536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:16.216317    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx" event={"ID":"43c7ca36-3d4d-496f-a3e8-67ec2e9b2243","Type":"ContainerDied","Data":"a7b103dc57a8aa3b61c1a5ffdc2af39de34e83c0bba59f50615890056d2b1e78"}
Oct 31 13:57:16.216536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:16.216386    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx" event={"ID":"43c7ca36-3d4d-496f-a3e8-67ec2e9b2243","Type":"ContainerDied","Data":"301d976ea6ea476295ae2cab33d0a90730fa320870f06e5e44827438b0508e37"}
Oct 31 13:57:16.216536 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:16.216417    4428 scope.go:122] "RemoveContainer" containerID="a7b103dc57a8aa3b61c1a5ffdc2af39de34e83c0bba59f50615890056d2b1e78"
Oct 31 13:57:16.231414 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:16.231393    4428 scope.go:122] "RemoveContainer" containerID="a7b103dc57a8aa3b61c1a5ffdc2af39de34e83c0bba59f50615890056d2b1e78"
Oct 31 13:57:16.231734 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:57:16.231715    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b103dc57a8aa3b61c1a5ffdc2af39de34e83c0bba59f50615890056d2b1e78\": container with ID starting with a7b103dc57a8aa3b61c1a5ffdc2af39de34e83c0bba59f50615890056d2b1e78 not found: ID does not exist" containerID="a7b103dc57a8aa3b61c1a5ffdc2af39de34e83c0bba59f50615890056d2b1e78"
Oct 31 13:57:16.231794 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:16.231743    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b103dc57a8aa3b61c1a5ffdc2af39de34e83c0bba59f50615890056d2b1e78"} err="failed to get container status \"a7b103dc57a8aa3b61c1a5ffdc2af39de34e83c0bba59f50615890056d2b1e78\": rpc error: code = NotFound desc = could not find container \"a7b103dc57a8aa3b61c1a5ffdc2af39de34e83c0bba59f50615890056d2b1e78\": container with ID starting with a7b103dc57a8aa3b61c1a5ffdc2af39de34e83c0bba59f50615890056d2b1e78 not found: ID does not exist"
Oct 31 13:57:16.243027 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:16.242998    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"]
Oct 31 13:57:16.250352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:16.250326    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50b22-5f4c877789-dwvbx"]
Oct 31 13:57:16.318444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:16.318416    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:16.758194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:16.758152    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c7ca36-3d4d-496f-a3e8-67ec2e9b2243" path="/var/lib/kubelet/pods/43c7ca36-3d4d-496f-a3e8-67ec2e9b2243/volumes"
Oct 31 13:57:17.000379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:17.000346    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:17.207514 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:17.207424    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:17.567898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:17.561566    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:17.729542 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:17.729493    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:18.087951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:18.087917    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:18.546895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:18.546711    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:18.746036 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:57:18.745997    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:57:19.435379 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:19.435326    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:20.251029 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:20.250978    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:21.217235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:21.217197    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:22.020896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:22.020817    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:22.892606 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:22.892565    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:23.707890 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:23.705108    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:24.583491 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:24.566582    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:25.277112 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:25.269504    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:25.437575 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:25.437536    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:25.648601 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:25.648451    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:26.094672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:26.094634    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:26.489239 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:26.489201    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:27.493076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:27.493035    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:28.113368 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:28.113336    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:28.290406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:28.290370    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:28.723064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:28.721570    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:29.266142 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:29.266104    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:31.012800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:31.012761    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"]
Oct 31 13:57:31.023259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:31.023223    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"]
Oct 31 13:57:31.685382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:31.685324    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:31.838931 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:31.838893    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:32.177599 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:32.177568    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:32.986751 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:32.986714    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:33.689625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:33.689589    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:33.853299 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:33.853262    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:34.024894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:34.024290    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:34.196785 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:34.196754    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:34.586254 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:34.586218    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:34.750056 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:34.750018    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:35.568336 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:35.568303    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:35.753607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:35.753554    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:36.159526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:36.159496    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:36.553337 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:36.553302    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"]
Oct 31 13:57:36.618477 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:36.618437    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:36.773348 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:36.773271    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:36.937418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:36.937328    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:37.797505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:37.797465    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:38.676560 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:38.676520    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:39.654460 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:39.654418    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:40.821738 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:40.821702    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:40.986023 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:40.985993    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:41.147434 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:41.147351    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:41.554540 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:41.554493    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5" podUID="c92cd891-8b41-4511-9fb0-d724cb1caaf5" containerName="payload-pre-processing" containerID="cri-o://ef85f40d2633f3795b91531f2493b1a6f20f80ce6592cbf79dd33e3063e35257" gracePeriod=25
Oct 31 13:57:41.870840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:41.870810    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"
Oct 31 13:57:41.874358 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:41.874330    4428 generic.go:356] "Generic (PLEG): container finished" podID="c92cd891-8b41-4511-9fb0-d724cb1caaf5" containerID="ef85f40d2633f3795b91531f2493b1a6f20f80ce6592cbf79dd33e3063e35257" exitCode=0
Oct 31 13:57:41.874513 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:41.874396    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"
Oct 31 13:57:41.874513 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:41.874429    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5" event={"ID":"c92cd891-8b41-4511-9fb0-d724cb1caaf5","Type":"ContainerDied","Data":"ef85f40d2633f3795b91531f2493b1a6f20f80ce6592cbf79dd33e3063e35257"}
Oct 31 13:57:41.874513 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:41.874480    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5" event={"ID":"c92cd891-8b41-4511-9fb0-d724cb1caaf5","Type":"ContainerDied","Data":"54e80c625804f599b2a54edd7eff7c1d76779570fe003c216b7202611a1ee2cc"}
Oct 31 13:57:41.874732 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:41.874514    4428 scope.go:122] "RemoveContainer" containerID="ef85f40d2633f3795b91531f2493b1a6f20f80ce6592cbf79dd33e3063e35257"
Oct 31 13:57:41.889091 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:41.889058    4428 scope.go:122] "RemoveContainer" containerID="ef85f40d2633f3795b91531f2493b1a6f20f80ce6592cbf79dd33e3063e35257"
Oct 31 13:57:41.889861 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:57:41.889812    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef85f40d2633f3795b91531f2493b1a6f20f80ce6592cbf79dd33e3063e35257\": container with ID starting with ef85f40d2633f3795b91531f2493b1a6f20f80ce6592cbf79dd33e3063e35257 not found: ID does not exist" containerID="ef85f40d2633f3795b91531f2493b1a6f20f80ce6592cbf79dd33e3063e35257"
Oct 31 13:57:41.890206 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:41.889856    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef85f40d2633f3795b91531f2493b1a6f20f80ce6592cbf79dd33e3063e35257"} err="failed to get container status \"ef85f40d2633f3795b91531f2493b1a6f20f80ce6592cbf79dd33e3063e35257\": rpc error: code = NotFound desc = could not find container \"ef85f40d2633f3795b91531f2493b1a6f20f80ce6592cbf79dd33e3063e35257\": container with ID starting with ef85f40d2633f3795b91531f2493b1a6f20f80ce6592cbf79dd33e3063e35257 not found: ID does not exist"
Oct 31 13:57:41.958059 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:41.958009    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:41.989650 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:41.989383    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c92cd891-8b41-4511-9fb0-d724cb1caaf5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92cd891-8b41-4511-9fb0-d724cb1caaf5-plugins-config-volume\") pod \"c92cd891-8b41-4511-9fb0-d724cb1caaf5\" (UID: \"c92cd891-8b41-4511-9fb0-d724cb1caaf5\") "
Oct 31 13:57:41.989855 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:41.989756    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c92cd891-8b41-4511-9fb0-d724cb1caaf5-kube-api-access-dwp8w\" (UniqueName: \"kubernetes.io/projected/c92cd891-8b41-4511-9fb0-d724cb1caaf5-kube-api-access-dwp8w\") pod \"c92cd891-8b41-4511-9fb0-d724cb1caaf5\" (UID: \"c92cd891-8b41-4511-9fb0-d724cb1caaf5\") "
Oct 31 13:57:42.015031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.014914    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c92cd891-8b41-4511-9fb0-d724cb1caaf5-plugins-config-volume" pod "c92cd891-8b41-4511-9fb0-d724cb1caaf5" (UID: "c92cd891-8b41-4511-9fb0-d724cb1caaf5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:57:42.020740 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.018983    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92cd891-8b41-4511-9fb0-d724cb1caaf5-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:42.058168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.058115    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92cd891-8b41-4511-9fb0-d724cb1caaf5-kube-api-access-dwp8w" pod "c92cd891-8b41-4511-9fb0-d724cb1caaf5" (UID: "c92cd891-8b41-4511-9fb0-d724cb1caaf5"). InnerVolumeSpecName "kube-api-access-dwp8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:57:42.116639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.116558    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"]
Oct 31 13:57:42.117411 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.117395    4428 cpu_manager.go:414] "containerMap: removing container" podUID="c92cd891-8b41-4511-9fb0-d724cb1caaf5" containerName="payload-pre-processing"
Oct 31 13:57:42.117469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.117414    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92cd891-8b41-4511-9fb0-d724cb1caaf5" containerName="payload-pre-processing"
Oct 31 13:57:42.117469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.117449    4428 cpu_manager.go:414] "containerMap: removing container" podUID="2e6f1d83-780e-4779-a9b6-fe0b967f149a" containerName="maas-api"
Oct 31 13:57:42.117469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.117454    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e6f1d83-780e-4779-a9b6-fe0b967f149a" containerName="maas-api"
Oct 31 13:57:42.117469 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.117468    4428 cpu_manager.go:414] "containerMap: removing container" podUID="43c7ca36-3d4d-496f-a3e8-67ec2e9b2243" containerName="payload-processing"
Oct 31 13:57:42.117605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.117474    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c7ca36-3d4d-496f-a3e8-67ec2e9b2243" containerName="payload-processing"
Oct 31 13:57:42.117605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.117593    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e6f1d83-780e-4779-a9b6-fe0b967f149a" containerName="maas-api"
Oct 31 13:57:42.117605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.117604    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c7ca36-3d4d-496f-a3e8-67ec2e9b2243" containerName="payload-processing"
Oct 31 13:57:42.117706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.117615    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92cd891-8b41-4511-9fb0-d724cb1caaf5" containerName="payload-pre-processing"
Oct 31 13:57:42.120406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.120376    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwp8w\" (UniqueName: \"kubernetes.io/projected/c92cd891-8b41-4511-9fb0-d724cb1caaf5-kube-api-access-dwp8w\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:42.123095 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.123060    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"
Oct 31 13:57:42.128300 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.128247    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9f9103\""
Oct 31 13:57:42.128977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.128953    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9f9103-dockercfg-f6qhh\""
Oct 31 13:57:42.130885 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.130498    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:42.131733 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.131708    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"]
Oct 31 13:57:42.169300 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.169265    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"]
Oct 31 13:57:42.174235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.174210    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"
Oct 31 13:57:42.186911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.185507    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"]
Oct 31 13:57:42.261895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.250497    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ad25ff-2a15-410b-8bb3-96633053d27d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689\" (UID: \"d4ad25ff-2a15-410b-8bb3-96633053d27d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"
Oct 31 13:57:42.261895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.250847    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9n2f\" (UniqueName: \"kubernetes.io/projected/d5d381df-edd5-406d-b348-1d0ffc55b612-kube-api-access-p9n2f\") pod \"payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4\" (UID: \"d5d381df-edd5-406d-b348-1d0ffc55b612\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"
Oct 31 13:57:42.261895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.251002    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhh2m\" (UniqueName: \"kubernetes.io/projected/d4ad25ff-2a15-410b-8bb3-96633053d27d-kube-api-access-fhh2m\") pod \"payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689\" (UID: \"d4ad25ff-2a15-410b-8bb3-96633053d27d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"
Oct 31 13:57:42.261895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.251055    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d381df-edd5-406d-b348-1d0ffc55b612-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4\" (UID: \"d5d381df-edd5-406d-b348-1d0ffc55b612\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"
Oct 31 13:57:42.316353 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.294971    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"]
Oct 31 13:57:42.347917 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.344387    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:42.368938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.365927    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ad25ff-2a15-410b-8bb3-96633053d27d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689\" (UID: \"d4ad25ff-2a15-410b-8bb3-96633053d27d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"
Oct 31 13:57:42.368938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.366204    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9n2f\" (UniqueName: \"kubernetes.io/projected/d5d381df-edd5-406d-b348-1d0ffc55b612-kube-api-access-p9n2f\") pod \"payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4\" (UID: \"d5d381df-edd5-406d-b348-1d0ffc55b612\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"
Oct 31 13:57:42.368938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.366313    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhh2m\" (UniqueName: \"kubernetes.io/projected/d4ad25ff-2a15-410b-8bb3-96633053d27d-kube-api-access-fhh2m\") pod \"payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689\" (UID: \"d4ad25ff-2a15-410b-8bb3-96633053d27d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"
Oct 31 13:57:42.368938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.366357    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d381df-edd5-406d-b348-1d0ffc55b612-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4\" (UID: \"d5d381df-edd5-406d-b348-1d0ffc55b612\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"
Oct 31 13:57:42.368938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.367125    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ad25ff-2a15-410b-8bb3-96633053d27d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689\" (UID: \"d4ad25ff-2a15-410b-8bb3-96633053d27d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"
Oct 31 13:57:42.368938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.367378    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d381df-edd5-406d-b348-1d0ffc55b612-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4\" (UID: \"d5d381df-edd5-406d-b348-1d0ffc55b612\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"
Oct 31 13:57:42.387718 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.376235    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5286ea-7fc5954f67-d8ws5"]
Oct 31 13:57:42.387718 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.387563    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhh2m\" (UniqueName: \"kubernetes.io/projected/d4ad25ff-2a15-410b-8bb3-96633053d27d-kube-api-access-fhh2m\") pod \"payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689\" (UID: \"d4ad25ff-2a15-410b-8bb3-96633053d27d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"
Oct 31 13:57:42.423361 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.413568    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9n2f\" (UniqueName: \"kubernetes.io/projected/d5d381df-edd5-406d-b348-1d0ffc55b612-kube-api-access-p9n2f\") pod \"payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4\" (UID: \"d5d381df-edd5-406d-b348-1d0ffc55b612\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"
Oct 31 13:57:42.444600 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.444296    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"
Oct 31 13:57:42.501226 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.501195    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"
Oct 31 13:57:42.762691 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.762651    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c92cd891-8b41-4511-9fb0-d724cb1caaf5" path="/var/lib/kubelet/pods/c92cd891-8b41-4511-9fb0-d724cb1caaf5/volumes"
Oct 31 13:57:42.805298 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.805264    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"]
Oct 31 13:57:42.806033 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:57:42.806011    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ad25ff_2a15_410b_8bb3_96633053d27d.slice/crio-8736a42d11980bd995a3bc09785678ddc9a198024862c66a371ba34cb6eb3aa4 WatchSource:0}: Error finding container 8736a42d11980bd995a3bc09785678ddc9a198024862c66a371ba34cb6eb3aa4: Status 404 returned error can't find the container with id 8736a42d11980bd995a3bc09785678ddc9a198024862c66a371ba34cb6eb3aa4
Oct 31 13:57:42.877133 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:57:42.877106    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d381df_edd5_406d_b348_1d0ffc55b612.slice/crio-1f128dc08bfea22c157aa5ba3e83702816225ba2a72ea2705519d4e486cc5950 WatchSource:0}: Error finding container 1f128dc08bfea22c157aa5ba3e83702816225ba2a72ea2705519d4e486cc5950: Status 404 returned error can't find the container with id 1f128dc08bfea22c157aa5ba3e83702816225ba2a72ea2705519d4e486cc5950
Oct 31 13:57:42.878807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.878779    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"]
Oct 31 13:57:42.886033 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:42.886005    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689" event={"ID":"d4ad25ff-2a15-410b-8bb3-96633053d27d","Type":"ContainerStarted","Data":"8736a42d11980bd995a3bc09785678ddc9a198024862c66a371ba34cb6eb3aa4"}
Oct 31 13:57:43.046294 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:43.046261    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:43.222263 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:43.221012    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:43.318712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:43.318674    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"]
Oct 31 13:57:43.318976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:43.318949    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td" podUID="4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" containerName="istio-proxy" containerID="cri-o://3642a44855090e9f67919838f85632847bace33c7926ae09901524c659c5ab72" gracePeriod=30
Oct 31 13:57:43.457894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:43.455489    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:43.904966 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:43.904924    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4" event={"ID":"d5d381df-edd5-406d-b348-1d0ffc55b612","Type":"ContainerStarted","Data":"705d3e8ef999eddd1ba32588b540f84ba6bf3dc124297db3a09aac68fcb3a257"}
Oct 31 13:57:43.905606 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:43.905572    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"
Oct 31 13:57:43.905762 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:43.905616    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4" event={"ID":"d5d381df-edd5-406d-b348-1d0ffc55b612","Type":"ContainerStarted","Data":"1f128dc08bfea22c157aa5ba3e83702816225ba2a72ea2705519d4e486cc5950"}
Oct 31 13:57:43.907532 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:43.907508    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689" event={"ID":"d4ad25ff-2a15-410b-8bb3-96633053d27d","Type":"ContainerStarted","Data":"2873947a16a03403208bdc29f3ac6f8931158ce8d913ce50fc8c9058e9c41064"}
Oct 31 13:57:43.907667 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:43.907637    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"
Oct 31 13:57:43.932172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:43.932116    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4" podStartSLOduration=1.9321015049999999 podStartE2EDuration="1.932101505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:57:43.926053297 +0000 UTC m=+3261.324401923" watchObservedRunningTime="2026-10-31 13:57:43.932101505 +0000 UTC m=+3261.330450129"
Oct 31 13:57:43.950999 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:43.950949    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689" podStartSLOduration=1.9509332160000001 podStartE2EDuration="1.950933216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:57:43.946417271 +0000 UTC m=+3261.344765897" watchObservedRunningTime="2026-10-31 13:57:43.950933216 +0000 UTC m=+3261.349281837"
Oct 31 13:57:44.038328 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:44.038286    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:57:44.038534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:44.038342    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td" podUID="4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:57:44.656894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:44.656408    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:45.753893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:45.751373    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:46.044970 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.044822    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main" containerID="cri-o://9dcfcf33b0a714bb03461ae1a28a0f14a6aa93649426c6e037567bb91e2a913d" gracePeriod=45
Oct 31 13:57:46.059601 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.059566    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main" containerID="cri-o://e630cb9ca71b25fdf7e82475339ac1b6d7f3d9ca4b1fb0cbd4af7f8a9796b66b" gracePeriod=45
Oct 31 13:57:46.359956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.359919    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:57:46.397032 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.396999    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:57:46.500477 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.500437    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-tmp-dir\") pod \"beb4582b-cb4b-437b-b527-6e77941a2543\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") "
Oct 31 13:57:46.500692 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.500502    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-model-cache\") pod \"ab6906eb-42bc-426f-9861-4e5af42cc54d\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") "
Oct 31 13:57:46.500692 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.500600    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-home\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-home\") pod \"beb4582b-cb4b-437b-b527-6e77941a2543\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") "
Oct 31 13:57:46.500692 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.500642    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab6906eb-42bc-426f-9861-4e5af42cc54d-kube-api-access-5lkh5\" (UniqueName: \"kubernetes.io/projected/ab6906eb-42bc-426f-9861-4e5af42cc54d-kube-api-access-5lkh5\") pod \"ab6906eb-42bc-426f-9861-4e5af42cc54d\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") "
Oct 31 13:57:46.500692 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.500675    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-home\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-home\") pod \"ab6906eb-42bc-426f-9861-4e5af42cc54d\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") "
Oct 31 13:57:46.501031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.500712    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab6906eb-42bc-426f-9861-4e5af42cc54d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab6906eb-42bc-426f-9861-4e5af42cc54d-tls-certs\") pod \"ab6906eb-42bc-426f-9861-4e5af42cc54d\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") "
Oct 31 13:57:46.501031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.500773    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-model-cache" pod "ab6906eb-42bc-426f-9861-4e5af42cc54d" (UID: "ab6906eb-42bc-426f-9861-4e5af42cc54d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:46.501031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.500780    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-tmp-dir" pod "beb4582b-cb4b-437b-b527-6e77941a2543" (UID: "beb4582b-cb4b-437b-b527-6e77941a2543"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:46.501031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.500817    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-home" pod "beb4582b-cb4b-437b-b527-6e77941a2543" (UID: "beb4582b-cb4b-437b-b527-6e77941a2543"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:46.501031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.500823    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-tmp-dir\") pod \"ab6906eb-42bc-426f-9861-4e5af42cc54d\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") "
Oct 31 13:57:46.501031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.500929    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-dshm\") pod \"beb4582b-cb4b-437b-b527-6e77941a2543\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") "
Oct 31 13:57:46.501031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.500962    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beb4582b-cb4b-437b-b527-6e77941a2543-tls-certs\" (UniqueName: \"kubernetes.io/secret/beb4582b-cb4b-437b-b527-6e77941a2543-tls-certs\") pod \"beb4582b-cb4b-437b-b527-6e77941a2543\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") "
Oct 31 13:57:46.501031 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.501018    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-model-cache\") pod \"beb4582b-cb4b-437b-b527-6e77941a2543\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") "
Oct 31 13:57:46.501505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.501059    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-dshm\") pod \"ab6906eb-42bc-426f-9861-4e5af42cc54d\" (UID: \"ab6906eb-42bc-426f-9861-4e5af42cc54d\") "
Oct 31 13:57:46.501505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.501155    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb4582b-cb4b-437b-b527-6e77941a2543-kube-api-access-w6lwm\" (UniqueName: \"kubernetes.io/projected/beb4582b-cb4b-437b-b527-6e77941a2543-kube-api-access-w6lwm\") pod \"beb4582b-cb4b-437b-b527-6e77941a2543\" (UID: \"beb4582b-cb4b-437b-b527-6e77941a2543\") "
Oct 31 13:57:46.501505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.501172    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-home" pod "ab6906eb-42bc-426f-9861-4e5af42cc54d" (UID: "ab6906eb-42bc-426f-9861-4e5af42cc54d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:46.501505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.501186    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-tmp-dir" pod "ab6906eb-42bc-426f-9861-4e5af42cc54d" (UID: "ab6906eb-42bc-426f-9861-4e5af42cc54d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:46.501505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.501436    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-model-cache" pod "beb4582b-cb4b-437b-b527-6e77941a2543" (UID: "beb4582b-cb4b-437b-b527-6e77941a2543"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:46.502015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.501817    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.502015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.501844    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.502015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.501858    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.502015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.501885    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.502015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.501898    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.502015 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.501913    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.504260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.504228    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb4582b-cb4b-437b-b527-6e77941a2543-kube-api-access-w6lwm" pod "beb4582b-cb4b-437b-b527-6e77941a2543" (UID: "beb4582b-cb4b-437b-b527-6e77941a2543"). InnerVolumeSpecName "kube-api-access-w6lwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:57:46.504438 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.504402    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-dshm" pod "beb4582b-cb4b-437b-b527-6e77941a2543" (UID: "beb4582b-cb4b-437b-b527-6e77941a2543"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:46.504597 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.504543    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab6906eb-42bc-426f-9861-4e5af42cc54d-kube-api-access-5lkh5" pod "ab6906eb-42bc-426f-9861-4e5af42cc54d" (UID: "ab6906eb-42bc-426f-9861-4e5af42cc54d"). InnerVolumeSpecName "kube-api-access-5lkh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:57:46.504880 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.504849    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab6906eb-42bc-426f-9861-4e5af42cc54d-tls-certs" pod "ab6906eb-42bc-426f-9861-4e5af42cc54d" (UID: "ab6906eb-42bc-426f-9861-4e5af42cc54d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:57:46.505027 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.505006    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beb4582b-cb4b-437b-b527-6e77941a2543-tls-certs" pod "beb4582b-cb4b-437b-b527-6e77941a2543" (UID: "beb4582b-cb4b-437b-b527-6e77941a2543"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:57:46.506542 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.506521    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-dshm" pod "ab6906eb-42bc-426f-9861-4e5af42cc54d" (UID: "ab6906eb-42bc-426f-9861-4e5af42cc54d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:46.603488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.603403    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb4582b-cb4b-437b-b527-6e77941a2543-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.603488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.603432    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb4582b-cb4b-437b-b527-6e77941a2543-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.603488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.603442    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab6906eb-42bc-426f-9861-4e5af42cc54d-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.603488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.603453    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6lwm\" (UniqueName: \"kubernetes.io/projected/beb4582b-cb4b-437b-b527-6e77941a2543-kube-api-access-w6lwm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.603488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.603462    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lkh5\" (UniqueName: \"kubernetes.io/projected/ab6906eb-42bc-426f-9861-4e5af42cc54d-kube-api-access-5lkh5\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.603488 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.603472    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab6906eb-42bc-426f-9861-4e5af42cc54d-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:46.941517 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.941411    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:46.962769 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.962729    4428 generic.go:356] "Generic (PLEG): container finished" podID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerID="e630cb9ca71b25fdf7e82475339ac1b6d7f3d9ca4b1fb0cbd4af7f8a9796b66b" exitCode=0
Oct 31 13:57:46.963038 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.962785    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" event={"ID":"ab6906eb-42bc-426f-9861-4e5af42cc54d","Type":"ContainerDied","Data":"e630cb9ca71b25fdf7e82475339ac1b6d7f3d9ca4b1fb0cbd4af7f8a9796b66b"}
Oct 31 13:57:46.963038 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.962891    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4" event={"ID":"ab6906eb-42bc-426f-9861-4e5af42cc54d","Type":"ContainerDied","Data":"a61f59d32064f4623391fdea4b723148176a8e951f3e1f7fc845779d95ccee82"}
Oct 31 13:57:46.963038 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.962926    4428 scope.go:122] "RemoveContainer" containerID="e630cb9ca71b25fdf7e82475339ac1b6d7f3d9ca4b1fb0cbd4af7f8a9796b66b"
Oct 31 13:57:46.963210 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.963145    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"
Oct 31 13:57:46.965435 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.965341    4428 generic.go:356] "Generic (PLEG): container finished" podID="beb4582b-cb4b-437b-b527-6e77941a2543" containerID="9dcfcf33b0a714bb03461ae1a28a0f14a6aa93649426c6e037567bb91e2a913d" exitCode=0
Oct 31 13:57:46.965435 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.965379    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"
Oct 31 13:57:46.965435 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.965386    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" event={"ID":"beb4582b-cb4b-437b-b527-6e77941a2543","Type":"ContainerDied","Data":"9dcfcf33b0a714bb03461ae1a28a0f14a6aa93649426c6e037567bb91e2a913d"}
Oct 31 13:57:46.965435 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.965415    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5" event={"ID":"beb4582b-cb4b-437b-b527-6e77941a2543","Type":"ContainerDied","Data":"cf981b441427c13bf8ebed73424a31d187e74c0dd95ea09386e2668053c072e2"}
Oct 31 13:57:46.979017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.978995    4428 scope.go:122] "RemoveContainer" containerID="e630cb9ca71b25fdf7e82475339ac1b6d7f3d9ca4b1fb0cbd4af7f8a9796b66b"
Oct 31 13:57:46.979369 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:57:46.979348    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e630cb9ca71b25fdf7e82475339ac1b6d7f3d9ca4b1fb0cbd4af7f8a9796b66b\": container with ID starting with e630cb9ca71b25fdf7e82475339ac1b6d7f3d9ca4b1fb0cbd4af7f8a9796b66b not found: ID does not exist" containerID="e630cb9ca71b25fdf7e82475339ac1b6d7f3d9ca4b1fb0cbd4af7f8a9796b66b"
Oct 31 13:57:46.979424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.979379    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e630cb9ca71b25fdf7e82475339ac1b6d7f3d9ca4b1fb0cbd4af7f8a9796b66b"} err="failed to get container status \"e630cb9ca71b25fdf7e82475339ac1b6d7f3d9ca4b1fb0cbd4af7f8a9796b66b\": rpc error: code = NotFound desc = could not find container \"e630cb9ca71b25fdf7e82475339ac1b6d7f3d9ca4b1fb0cbd4af7f8a9796b66b\": container with ID starting with e630cb9ca71b25fdf7e82475339ac1b6d7f3d9ca4b1fb0cbd4af7f8a9796b66b not found: ID does not exist"
Oct 31 13:57:46.979470 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:46.979427    4428 scope.go:122] "RemoveContainer" containerID="9dcfcf33b0a714bb03461ae1a28a0f14a6aa93649426c6e037567bb91e2a913d"
Oct 31 13:57:47.006598 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.006571    4428 scope.go:122] "RemoveContainer" containerID="9dcfcf33b0a714bb03461ae1a28a0f14a6aa93649426c6e037567bb91e2a913d"
Oct 31 13:57:47.007798 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:57:47.007527    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dcfcf33b0a714bb03461ae1a28a0f14a6aa93649426c6e037567bb91e2a913d\": container with ID starting with 9dcfcf33b0a714bb03461ae1a28a0f14a6aa93649426c6e037567bb91e2a913d not found: ID does not exist" containerID="9dcfcf33b0a714bb03461ae1a28a0f14a6aa93649426c6e037567bb91e2a913d"
Oct 31 13:57:47.007798 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.007564    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dcfcf33b0a714bb03461ae1a28a0f14a6aa93649426c6e037567bb91e2a913d"} err="failed to get container status \"9dcfcf33b0a714bb03461ae1a28a0f14a6aa93649426c6e037567bb91e2a913d\": rpc error: code = NotFound desc = could not find container \"9dcfcf33b0a714bb03461ae1a28a0f14a6aa93649426c6e037567bb91e2a913d\": container with ID starting with 9dcfcf33b0a714bb03461ae1a28a0f14a6aa93649426c6e037567bb91e2a913d not found: ID does not exist"
Oct 31 13:57:47.007798 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.007739    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"]
Oct 31 13:57:47.012607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.012580    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/premium-w2-5286ea-kserve-649fcb656f-gr7p5"]
Oct 31 13:57:47.017954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.017929    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"]
Oct 31 13:57:47.022423 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.022402    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-5286ea/simulator-w2-5286ea-kserve-6455577db4-w6gf4"]
Oct 31 13:57:47.643391 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.643334    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:47.913791 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.913700    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:47.961651 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.961614    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"]
Oct 31 13:57:47.962712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.962688    4428 cpu_manager.go:414] "containerMap: removing container" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main"
Oct 31 13:57:47.962712 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.962711    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main"
Oct 31 13:57:47.962975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.962761    4428 cpu_manager.go:414] "containerMap: removing container" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main"
Oct 31 13:57:47.962975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.962770    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main"
Oct 31 13:57:47.963088 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.963029    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" containerName="main"
Oct 31 13:57:47.963088 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.963060    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" containerName="main"
Oct 31 13:57:47.968460 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.968436    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:47.976653 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.976622    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"]
Oct 31 13:57:47.979028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:47.979004    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5af6eb-openshift-default-dockercfg-tw7lr\""
Oct 31 13:57:48.350823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.342235    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-data\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.350823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.342345    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-token\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.350823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.342395    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-credential-socket\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.350823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.342429    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-envoy\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.350823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.342504    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-podinfo\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.350823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.342613    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z9jv\" (UniqueName: \"kubernetes.io/projected/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-kube-api-access-2z9jv\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.350823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.342809    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-workload-certs\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.350823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.342941    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istiod-ca-cert\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.350823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.343024    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-workload-socket\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.350823 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.344327    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:48.444302 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.444267    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istiod-ca-cert\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.444442 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.444341    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-workload-socket\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.444442 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.444385    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-data\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.444719 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.444462    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-token\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.444719 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.444502    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-credential-socket\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.444719 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.444537    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-envoy\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.444719 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.444610    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-podinfo\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.444719 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.444707    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9jv\" (UniqueName: \"kubernetes.io/projected/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-kube-api-access-2z9jv\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.445860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.444913    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-workload-certs\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.445860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.445112    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istiod-ca-cert\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.445860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.445307    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-workload-certs\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.445860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.445329    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-data\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.446481 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.446458    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-credential-socket\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.446620 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.446599    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-workload-socket\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.448681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.448656    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-podinfo\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.448776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.448737    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-envoy\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.458826 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.458797    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9jv\" (UniqueName: \"kubernetes.io/projected/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-kube-api-access-2z9jv\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.459567 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.459543    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-token\") pod \"e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") " pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.502338 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.502302    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:48.588977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.588941    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:48.647538 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.647508    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:57:48.745095 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:57:48.745057    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:57:48.749266 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.749238    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-credential-socket\") pod \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") "
Oct 31 13:57:48.749456 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.749434    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-podinfo\") pod \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") "
Oct 31 13:57:48.749532 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.749479    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-token\" (UniqueName: \"kubernetes.io/projected/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-token\") pod \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") "
Oct 31 13:57:48.749598 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.749527    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-data\") pod \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") "
Oct 31 13:57:48.749598 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.749538    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-credential-socket" pod "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" (UID: "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:48.749716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.749612    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istiod-ca-cert\") pod \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") "
Oct 31 13:57:48.749716 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.749661    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-envoy\") pod \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") "
Oct 31 13:57:48.749834 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.749767    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-kube-api-access-49p98\" (UniqueName: \"kubernetes.io/projected/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-kube-api-access-49p98\") pod \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") "
Oct 31 13:57:48.749834 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.749803    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-workload-certs\") pod \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") "
Oct 31 13:57:48.749834 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.749807    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-data" pod "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" (UID: "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:48.750091 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.749904    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-workload-socket\") pod \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\" (UID: \"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86\") "
Oct 31 13:57:48.750523 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.750497    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:48.750625 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.750528    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:48.750740 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.750720    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-workload-socket" pod "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" (UID: "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:48.750977 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.750951    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istiod-ca-cert" pod "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" (UID: "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:57:48.751063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.751033    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-workload-certs" pod "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" (UID: "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:48.752574 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.752535    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-podinfo" pod "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" (UID: "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:57:48.753369 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.753342    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-token" pod "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" (UID: "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:57:48.753564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.753525    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-kube-api-access-49p98" pod "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" (UID: "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86"). InnerVolumeSpecName "kube-api-access-49p98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:57:48.753814 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.753791    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-envoy" pod "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" (UID: "4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:57:48.761161 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.761131    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab6906eb-42bc-426f-9861-4e5af42cc54d" path="/var/lib/kubelet/pods/ab6906eb-42bc-426f-9861-4e5af42cc54d/volumes"
Oct 31 13:57:48.761863 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.761842    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb4582b-cb4b-437b-b527-6e77941a2543" path="/var/lib/kubelet/pods/beb4582b-cb4b-437b-b527-6e77941a2543/volumes"
Oct 31 13:57:48.852760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.852723    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:48.852760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.852761    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:48.853149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.852778    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:48.853149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.852792    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:48.853149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.852805    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49p98\" (UniqueName: \"kubernetes.io/projected/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-kube-api-access-49p98\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:48.853149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.852817    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:48.853149 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.852829    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:57:48.888553 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.888522    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"]
Oct 31 13:57:48.889577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:57:48.889546    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6991ec79_5e5e_4a07_b3c8_dffe08f0d42d.slice/crio-7056740e8bb7367167c27982d4d3fe28d04e495cb82389603b2a658db4d460a1 WatchSource:0}: Error finding container 7056740e8bb7367167c27982d4d3fe28d04e495cb82389603b2a658db4d460a1: Status 404 returned error can't find the container with id 7056740e8bb7367167c27982d4d3fe28d04e495cb82389603b2a658db4d460a1
Oct 31 13:57:48.892630 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.892594    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:57:48.892720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.892666    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:57:48.892720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:48.892694    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:57:49.015907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.012729    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:49.053386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.053268    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj" event={"ID":"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d","Type":"ContainerStarted","Data":"460cfc5b72ecfb55a6b52dbb7b554542b0791c37027e4312fe093a7db81604ef"}
Oct 31 13:57:49.053386 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.053347    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj" event={"ID":"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d","Type":"ContainerStarted","Data":"7056740e8bb7367167c27982d4d3fe28d04e495cb82389603b2a658db4d460a1"}
Oct 31 13:57:49.056821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.056790    4428 generic.go:356] "Generic (PLEG): container finished" podID="4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" containerID="3642a44855090e9f67919838f85632847bace33c7926ae09901524c659c5ab72" exitCode=0
Oct 31 13:57:49.057041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.056936    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td" event={"ID":"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86","Type":"ContainerDied","Data":"3642a44855090e9f67919838f85632847bace33c7926ae09901524c659c5ab72"}
Oct 31 13:57:49.057041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.056972    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td" event={"ID":"4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86","Type":"ContainerDied","Data":"3bec6d07b2ddbeea37017005d03a4387436ae72ad8d2e502469f4a86a2927323"}
Oct 31 13:57:49.057041 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.056995    4428 scope.go:122] "RemoveContainer" containerID="3642a44855090e9f67919838f85632847bace33c7926ae09901524c659c5ab72"
Oct 31 13:57:49.057228 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.057103    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"
Oct 31 13:57:49.078919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.077699    4428 scope.go:122] "RemoveContainer" containerID="af47517c0baeef3ceae69ac5f69521d0973bbf65792ccb61ce7f17c82ac5d84e"
Oct 31 13:57:49.082480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.082423    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj" podStartSLOduration=2.082410588 podStartE2EDuration="2.082410588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:57:49.078189931 +0000 UTC m=+3266.476538593" watchObservedRunningTime="2026-10-31 13:57:49.082410588 +0000 UTC m=+3266.480759211"
Oct 31 13:57:49.091740 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.091698    4428 scope.go:122] "RemoveContainer" containerID="3642a44855090e9f67919838f85632847bace33c7926ae09901524c659c5ab72"
Oct 31 13:57:49.092763 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:57:49.092517    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3642a44855090e9f67919838f85632847bace33c7926ae09901524c659c5ab72\": container with ID starting with 3642a44855090e9f67919838f85632847bace33c7926ae09901524c659c5ab72 not found: ID does not exist" containerID="3642a44855090e9f67919838f85632847bace33c7926ae09901524c659c5ab72"
Oct 31 13:57:49.092763 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.092559    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3642a44855090e9f67919838f85632847bace33c7926ae09901524c659c5ab72"} err="failed to get container status \"3642a44855090e9f67919838f85632847bace33c7926ae09901524c659c5ab72\": rpc error: code = NotFound desc = could not find container \"3642a44855090e9f67919838f85632847bace33c7926ae09901524c659c5ab72\": container with ID starting with 3642a44855090e9f67919838f85632847bace33c7926ae09901524c659c5ab72 not found: ID does not exist"
Oct 31 13:57:49.092763 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.092585    4428 scope.go:122] "RemoveContainer" containerID="af47517c0baeef3ceae69ac5f69521d0973bbf65792ccb61ce7f17c82ac5d84e"
Oct 31 13:57:49.093077 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:57:49.092814    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af47517c0baeef3ceae69ac5f69521d0973bbf65792ccb61ce7f17c82ac5d84e\": container with ID starting with af47517c0baeef3ceae69ac5f69521d0973bbf65792ccb61ce7f17c82ac5d84e not found: ID does not exist" containerID="af47517c0baeef3ceae69ac5f69521d0973bbf65792ccb61ce7f17c82ac5d84e"
Oct 31 13:57:49.093077 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.092833    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af47517c0baeef3ceae69ac5f69521d0973bbf65792ccb61ce7f17c82ac5d84e"} err="failed to get container status \"af47517c0baeef3ceae69ac5f69521d0973bbf65792ccb61ce7f17c82ac5d84e\": rpc error: code = NotFound desc = could not find container \"af47517c0baeef3ceae69ac5f69521d0973bbf65792ccb61ce7f17c82ac5d84e\": container with ID starting with af47517c0baeef3ceae69ac5f69521d0973bbf65792ccb61ce7f17c82ac5d84e not found: ID does not exist"
Oct 31 13:57:49.098191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.098167    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"]
Oct 31 13:57:49.103080 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.103059    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-dafbce-openshift-default-6fb8c547ff-2d7td"]
Oct 31 13:57:49.589558 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.589496    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:49.904894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:49.901442    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:50.596397 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:50.596363    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:50.758457 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:50.758378    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" path="/var/lib/kubelet/pods/4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86/volumes"
Oct 31 13:57:50.828027 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:50.823481    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:51.073485 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:51.073453    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:51.074624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:51.074606    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:57:51.419592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:51.419505    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:51.576127 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:51.576098    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:51.858367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:51.858011    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:52.646005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:52.645939    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:52.675444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:52.675407    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"]
Oct 31 13:57:52.676263 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:52.676248    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" containerName="istio-proxy"
Oct 31 13:57:52.676331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:52.676265    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" containerName="istio-proxy"
Oct 31 13:57:52.676406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:52.676396    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" containerName="istio-proxy"
Oct 31 13:57:52.676448 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:52.676410    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" containerName="istio-proxy"
Oct 31 13:57:52.680510 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:52.680493    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"
Oct 31 13:57:52.685579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:52.685554    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5af6eb-dockercfg-ltcsg\""
Oct 31 13:57:52.685839 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:52.685572    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5af6eb\""
Oct 31 13:57:52.691196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:52.691171    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"]
Oct 31 13:57:52.748789 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:52.748750    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq4q7\" (UniqueName: \"kubernetes.io/projected/e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0-kube-api-access-kq4q7\") pod \"payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2\" (UID: \"e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"
Oct 31 13:57:52.749030 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:52.748990    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2\" (UID: \"e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"
Oct 31 13:57:52.893893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:52.893776    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:52.942025 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:52.898386    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq4q7\" (UniqueName: \"kubernetes.io/projected/e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0-kube-api-access-kq4q7\") pod \"payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2\" (UID: \"e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"
Oct 31 13:57:52.942025 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:52.898551    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2\" (UID: \"e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"
Oct 31 13:57:52.942025 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:52.901714    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2\" (UID: \"e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"
Oct 31 13:57:52.968529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:52.968494    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq4q7\" (UniqueName: \"kubernetes.io/projected/e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0-kube-api-access-kq4q7\") pod \"payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2\" (UID: \"e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"
Oct 31 13:57:53.000774 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:53.000481    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"
Oct 31 13:57:53.102781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:53.102747    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:53.430419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:53.430385    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"]
Oct 31 13:57:53.431436 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:57:53.431408    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3aa47ef_57eb_4f88_ba2c_d07f9ed188c0.slice/crio-7ba908e2b8afd3b4c55e42f9d44fb15fa74e25e17048b92a2026ebb145e1dcea WatchSource:0}: Error finding container 7ba908e2b8afd3b4c55e42f9d44fb15fa74e25e17048b92a2026ebb145e1dcea: Status 404 returned error can't find the container with id 7ba908e2b8afd3b4c55e42f9d44fb15fa74e25e17048b92a2026ebb145e1dcea
Oct 31 13:57:53.809892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:53.807229    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:53.980904 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:53.980828    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"]
Oct 31 13:57:53.996211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:53.996176    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"]
Oct 31 13:57:54.113628 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:54.113529    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:54.159471 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:54.159432    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2" event={"ID":"e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0","Type":"ContainerStarted","Data":"583105638b068265a65e08653802d465ecc11e1c2684e4598bf1a4076ddd0c2e"}
Oct 31 13:57:54.159695 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:54.159491    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2" event={"ID":"e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0","Type":"ContainerStarted","Data":"7ba908e2b8afd3b4c55e42f9d44fb15fa74e25e17048b92a2026ebb145e1dcea"}
Oct 31 13:57:54.159695 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:54.159525    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"
Oct 31 13:57:54.183250 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:54.183177    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2" podStartSLOduration=2.183155501 podStartE2EDuration="2.183155501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:57:54.180951347 +0000 UTC m=+3271.579299971" watchObservedRunningTime="2026-10-31 13:57:54.183155501 +0000 UTC m=+3271.581504127"
Oct 31 13:57:54.268755 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:54.268724    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:54.747898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:54.744339    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:54.918770 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:54.918742    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"
Oct 31 13:57:54.919017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:54.918808    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"
Oct 31 13:57:55.704892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:55.696196    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:56.463558 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:56.463509    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:57.217107 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:57.217069    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:58.196894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:58.193467    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:57:58.986941 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:58.984949    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96" podUID="c622d692-1344-4482-9c4c-c26ae52a6642" containerName="payload-processing" containerID="cri-o://8c5defb178a8e6c27c18b9978d94f82850c367db196518a2a2f102b132004c9e" gracePeriod=25
Oct 31 13:57:59.007006 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:59.006954    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj" podUID="67ee4cb2-a701-44af-a9d3-538ef85c939c" containerName="payload-pre-processing" containerID="cri-o://1413b9af9952455389618bcd762edca94c7c64246f3dae1aa8407cee48686dcb" gracePeriod=25
Oct 31 13:57:59.434949 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:57:59.434403    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:00.593899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:00.593576    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:01.458896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:01.455469    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:02.263906 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:02.261489    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:03.077170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.077132    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"]
Oct 31 13:58:03.078355 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.078330    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" containerName="istio-proxy"
Oct 31 13:58:03.078355 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.078353    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e427f0a-ca46-4bc6-ad3b-9045d6ac9b86" containerName="istio-proxy"
Oct 31 13:58:03.083022 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.082995    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 13:58:03.092145 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.092108    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-9f9103-serving-cert\""
Oct 31 13:58:03.094576 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.094546    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"]
Oct 31 13:58:03.181510 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.181475    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:03.195418 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.195386    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4059a5cf-e441-4c84-9f57-e237f45a853f-metrics-certs\") pod \"maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv\" (UID: \"4059a5cf-e441-4c84-9f57-e237f45a853f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 13:58:03.195578 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.195441    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4059a5cf-e441-4c84-9f57-e237f45a853f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv\" (UID: \"4059a5cf-e441-4c84-9f57-e237f45a853f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 13:58:03.195650 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.195633    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4059a5cf-e441-4c84-9f57-e237f45a853f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv\" (UID: \"4059a5cf-e441-4c84-9f57-e237f45a853f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 13:58:03.195774 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.195754    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvhv8\" (UniqueName: \"kubernetes.io/projected/4059a5cf-e441-4c84-9f57-e237f45a853f-kube-api-access-cvhv8\") pod \"maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv\" (UID: \"4059a5cf-e441-4c84-9f57-e237f45a853f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 13:58:03.297788 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.297755    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4059a5cf-e441-4c84-9f57-e237f45a853f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv\" (UID: \"4059a5cf-e441-4c84-9f57-e237f45a853f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 13:58:03.298371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.297944    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4059a5cf-e441-4c84-9f57-e237f45a853f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv\" (UID: \"4059a5cf-e441-4c84-9f57-e237f45a853f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 13:58:03.298371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.298043    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvhv8\" (UniqueName: \"kubernetes.io/projected/4059a5cf-e441-4c84-9f57-e237f45a853f-kube-api-access-cvhv8\") pod \"maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv\" (UID: \"4059a5cf-e441-4c84-9f57-e237f45a853f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 13:58:03.298371 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.298175    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4059a5cf-e441-4c84-9f57-e237f45a853f-metrics-certs\") pod \"maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv\" (UID: \"4059a5cf-e441-4c84-9f57-e237f45a853f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 13:58:03.298689 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.298662    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4059a5cf-e441-4c84-9f57-e237f45a853f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv\" (UID: \"4059a5cf-e441-4c84-9f57-e237f45a853f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 13:58:03.300995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.300971    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4059a5cf-e441-4c84-9f57-e237f45a853f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv\" (UID: \"4059a5cf-e441-4c84-9f57-e237f45a853f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 13:58:03.301172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.301064    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4059a5cf-e441-4c84-9f57-e237f45a853f-metrics-certs\") pod \"maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv\" (UID: \"4059a5cf-e441-4c84-9f57-e237f45a853f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 13:58:03.311986 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.311958    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvhv8\" (UniqueName: \"kubernetes.io/projected/4059a5cf-e441-4c84-9f57-e237f45a853f-kube-api-access-cvhv8\") pod \"maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv\" (UID: \"4059a5cf-e441-4c84-9f57-e237f45a853f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 13:58:03.343167 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.343081    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:03.407654 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.407625    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 13:58:03.758283 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.756420    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"]
Oct 31 13:58:03.760207 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:58:03.760178    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4059a5cf_e441_4c84_9f57_e237f45a853f.slice/crio-9e937ea0f773cf7d60b0d550f4bfcd7ae5a5f25c77de3ac542952e29aea9c765 WatchSource:0}: Error finding container 9e937ea0f773cf7d60b0d550f4bfcd7ae5a5f25c77de3ac542952e29aea9c765: Status 404 returned error can't find the container with id 9e937ea0f773cf7d60b0d550f4bfcd7ae5a5f25c77de3ac542952e29aea9c765
Oct 31 13:58:03.761679 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:03.761657    4428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:58:04.204196 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:04.204008    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:04.341942 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:04.341911    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv" event={"ID":"4059a5cf-e441-4c84-9f57-e237f45a853f","Type":"ContainerStarted","Data":"9e937ea0f773cf7d60b0d550f4bfcd7ae5a5f25c77de3ac542952e29aea9c765"}
Oct 31 13:58:04.916250 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:04.916203    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:05.105440 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:05.105404    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:05.247210 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:05.229639    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"
Oct 31 13:58:05.290444 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:05.290254    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:05.403492 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:05.403454    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv" event={"ID":"4059a5cf-e441-4c84-9f57-e237f45a853f","Type":"ContainerStarted","Data":"e1a3b8b1c64f7cc6967cfde0e4bb28b4dbea8a06bcf05613d982345245a4585d"}
Oct 31 13:58:05.404040 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:05.403833    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 13:58:05.432811 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:05.432699    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv" podStartSLOduration=1.894500738 podStartE2EDuration="2.432679906s" totalImagesPullingTime="538.179168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:58:05.426984506 +0000 UTC m=+3282.825333131" watchObservedRunningTime="2026-10-31 13:58:05.432679906 +0000 UTC m=+3282.831028530"
Oct 31 13:58:05.841695 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:05.841654    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:06.009564 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:06.009525    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:06.826831 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:06.823160    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:07.561168 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:07.561129    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:07.804820 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:07.804776    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:08.021901 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:08.017564    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:08.212572 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:08.212493    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:08.748262 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:08.748221    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:08.853596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:08.853555    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96" podUID="c622d692-1344-4482-9c4c-c26ae52a6642" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:58:08.853596 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:08.853575    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj" podUID="67ee4cb2-a701-44af-a9d3-538ef85c939c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:58:09.316821 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:09.316772    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:10.196896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:10.193755    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:11.068104 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:11.068057    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:11.299781 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:11.299749    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:11.419846 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:11.419768    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 13:58:11.637410 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:11.637369    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"]
Oct 31 13:58:11.637945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:11.637909    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l" podUID="b844988a-dac6-463f-898b-ec37e3447ff1" containerName="maas-api" containerID="cri-o://2018b06c0d8e27bd42a7be51b1f078af63bc98a8f2984ada65e440e02c27fb02" gracePeriod=30
Oct 31 13:58:11.736768 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:11.736736    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:11.987959 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:11.975809    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:12.152512 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.150040    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:12.195475 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.192665    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:58:12.356213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.356118    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b844988a-dac6-463f-898b-ec37e3447ff1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b844988a-dac6-463f-898b-ec37e3447ff1-metrics-certs\") pod \"b844988a-dac6-463f-898b-ec37e3447ff1\" (UID: \"b844988a-dac6-463f-898b-ec37e3447ff1\") "
Oct 31 13:58:12.356385 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.356337    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b844988a-dac6-463f-898b-ec37e3447ff1-kube-api-access-ggj99\" (UniqueName: \"kubernetes.io/projected/b844988a-dac6-463f-898b-ec37e3447ff1-kube-api-access-ggj99\") pod \"b844988a-dac6-463f-898b-ec37e3447ff1\" (UID: \"b844988a-dac6-463f-898b-ec37e3447ff1\") "
Oct 31 13:58:12.356476 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.356392    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b844988a-dac6-463f-898b-ec37e3447ff1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b844988a-dac6-463f-898b-ec37e3447ff1-maas-api-tls\") pod \"b844988a-dac6-463f-898b-ec37e3447ff1\" (UID: \"b844988a-dac6-463f-898b-ec37e3447ff1\") "
Oct 31 13:58:12.356537 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.356511    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b844988a-dac6-463f-898b-ec37e3447ff1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b844988a-dac6-463f-898b-ec37e3447ff1-openshift-service-ca\") pod \"b844988a-dac6-463f-898b-ec37e3447ff1\" (UID: \"b844988a-dac6-463f-898b-ec37e3447ff1\") "
Oct 31 13:58:12.356899 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.356845    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b844988a-dac6-463f-898b-ec37e3447ff1-openshift-service-ca" pod "b844988a-dac6-463f-898b-ec37e3447ff1" (UID: "b844988a-dac6-463f-898b-ec37e3447ff1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:58:12.357277 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.357256    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b844988a-dac6-463f-898b-ec37e3447ff1-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:12.358707 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.358684    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b844988a-dac6-463f-898b-ec37e3447ff1-maas-api-tls" pod "b844988a-dac6-463f-898b-ec37e3447ff1" (UID: "b844988a-dac6-463f-898b-ec37e3447ff1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:58:12.359215 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.359193    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b844988a-dac6-463f-898b-ec37e3447ff1-metrics-certs" pod "b844988a-dac6-463f-898b-ec37e3447ff1" (UID: "b844988a-dac6-463f-898b-ec37e3447ff1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:58:12.359304 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.359221    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b844988a-dac6-463f-898b-ec37e3447ff1-kube-api-access-ggj99" pod "b844988a-dac6-463f-898b-ec37e3447ff1" (UID: "b844988a-dac6-463f-898b-ec37e3447ff1"). InnerVolumeSpecName "kube-api-access-ggj99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:12.456722 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.456667    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:12.462257 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.462218    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggj99\" (UniqueName: \"kubernetes.io/projected/b844988a-dac6-463f-898b-ec37e3447ff1-kube-api-access-ggj99\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:12.462257 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.462262    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b844988a-dac6-463f-898b-ec37e3447ff1-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:12.462491 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.462277    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b844988a-dac6-463f-898b-ec37e3447ff1-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:12.475010 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.474967    4428 generic.go:356] "Generic (PLEG): container finished" podID="b844988a-dac6-463f-898b-ec37e3447ff1" containerID="2018b06c0d8e27bd42a7be51b1f078af63bc98a8f2984ada65e440e02c27fb02" exitCode=0
Oct 31 13:58:12.475235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.475037    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"
Oct 31 13:58:12.475235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.475060    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l" event={"ID":"b844988a-dac6-463f-898b-ec37e3447ff1","Type":"ContainerDied","Data":"2018b06c0d8e27bd42a7be51b1f078af63bc98a8f2984ada65e440e02c27fb02"}
Oct 31 13:58:12.475235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.475116    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l" event={"ID":"b844988a-dac6-463f-898b-ec37e3447ff1","Type":"ContainerDied","Data":"2dc3f0be25e79c64cbb6cfa9c4e05950cd23f2c85335e714e082f9a0c68b176d"}
Oct 31 13:58:12.475235 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.475183    4428 scope.go:122] "RemoveContainer" containerID="2018b06c0d8e27bd42a7be51b1f078af63bc98a8f2984ada65e440e02c27fb02"
Oct 31 13:58:12.491710 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.491687    4428 scope.go:122] "RemoveContainer" containerID="2018b06c0d8e27bd42a7be51b1f078af63bc98a8f2984ada65e440e02c27fb02"
Oct 31 13:58:12.492109 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:58:12.492072    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2018b06c0d8e27bd42a7be51b1f078af63bc98a8f2984ada65e440e02c27fb02\": container with ID starting with 2018b06c0d8e27bd42a7be51b1f078af63bc98a8f2984ada65e440e02c27fb02 not found: ID does not exist" containerID="2018b06c0d8e27bd42a7be51b1f078af63bc98a8f2984ada65e440e02c27fb02"
Oct 31 13:58:12.492204 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.492107    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2018b06c0d8e27bd42a7be51b1f078af63bc98a8f2984ada65e440e02c27fb02"} err="failed to get container status \"2018b06c0d8e27bd42a7be51b1f078af63bc98a8f2984ada65e440e02c27fb02\": rpc error: code = NotFound desc = could not find container \"2018b06c0d8e27bd42a7be51b1f078af63bc98a8f2984ada65e440e02c27fb02\": container with ID starting with 2018b06c0d8e27bd42a7be51b1f078af63bc98a8f2984ada65e440e02c27fb02 not found: ID does not exist"
Oct 31 13:58:12.512416 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.512387    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"]
Oct 31 13:58:12.519424 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.519396    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6a3533-f75459bb7-56g4l"]
Oct 31 13:58:12.741750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.741717    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:12.760265 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.760232    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b844988a-dac6-463f-898b-ec37e3447ff1" path="/var/lib/kubelet/pods/b844988a-dac6-463f-898b-ec37e3447ff1/volumes"
Oct 31 13:58:12.910749 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:12.910721    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:13.284851 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:13.284818    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:13.476894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:13.473645    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:13.644971 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:13.644397    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:14.613893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:14.612269    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:15.186076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:15.186044    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:15.409897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:15.407635    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:16.181928 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:16.179119    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:16.615905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:16.607824    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:17.420677 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:17.416895    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:18.011825 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:18.011794    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:18.281223 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:18.281134    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:18.761884 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:58:18.761833    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:58:18.853193 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:18.853155    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj" podUID="67ee4cb2-a701-44af-a9d3-538ef85c939c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:58:18.853409 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:18.853159    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96" podUID="c622d692-1344-4482-9c4c-c26ae52a6642" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:58:19.181976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.181671    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:19.280046 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.280004    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"
Oct 31 13:58:19.365923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.363399    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c622d692-1344-4482-9c4c-c26ae52a6642-kube-api-access-r7pf9\" (UniqueName: \"kubernetes.io/projected/c622d692-1344-4482-9c4c-c26ae52a6642-kube-api-access-r7pf9\") pod \"c622d692-1344-4482-9c4c-c26ae52a6642\" (UID: \"c622d692-1344-4482-9c4c-c26ae52a6642\") "
Oct 31 13:58:19.365923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.363507    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c622d692-1344-4482-9c4c-c26ae52a6642-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c622d692-1344-4482-9c4c-c26ae52a6642-plugins-config-volume\") pod \"c622d692-1344-4482-9c4c-c26ae52a6642\" (UID: \"c622d692-1344-4482-9c4c-c26ae52a6642\") "
Oct 31 13:58:19.365923 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.365317    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c622d692-1344-4482-9c4c-c26ae52a6642-plugins-config-volume" pod "c622d692-1344-4482-9c4c-c26ae52a6642" (UID: "c622d692-1344-4482-9c4c-c26ae52a6642"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:58:19.366804 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.366772    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c622d692-1344-4482-9c4c-c26ae52a6642-kube-api-access-r7pf9" pod "c622d692-1344-4482-9c4c-c26ae52a6642" (UID: "c622d692-1344-4482-9c4c-c26ae52a6642"). InnerVolumeSpecName "kube-api-access-r7pf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:19.389744 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.389716    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"
Oct 31 13:58:19.466707 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.466663    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c622d692-1344-4482-9c4c-c26ae52a6642-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:19.466707 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.466699    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7pf9\" (UniqueName: \"kubernetes.io/projected/c622d692-1344-4482-9c4c-c26ae52a6642-kube-api-access-r7pf9\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:19.567633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.567540    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67ee4cb2-a701-44af-a9d3-538ef85c939c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ee4cb2-a701-44af-a9d3-538ef85c939c-plugins-config-volume\") pod \"67ee4cb2-a701-44af-a9d3-538ef85c939c\" (UID: \"67ee4cb2-a701-44af-a9d3-538ef85c939c\") "
Oct 31 13:58:19.567799 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.567641    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ee4cb2-a701-44af-a9d3-538ef85c939c-kube-api-access-ztfvd\" (UniqueName: \"kubernetes.io/projected/67ee4cb2-a701-44af-a9d3-538ef85c939c-kube-api-access-ztfvd\") pod \"67ee4cb2-a701-44af-a9d3-538ef85c939c\" (UID: \"67ee4cb2-a701-44af-a9d3-538ef85c939c\") "
Oct 31 13:58:19.567975 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.567953    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ee4cb2-a701-44af-a9d3-538ef85c939c-plugins-config-volume" pod "67ee4cb2-a701-44af-a9d3-538ef85c939c" (UID: "67ee4cb2-a701-44af-a9d3-538ef85c939c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:58:19.568253 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.568236    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ee4cb2-a701-44af-a9d3-538ef85c939c-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:19.570134 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.570110    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ee4cb2-a701-44af-a9d3-538ef85c939c-kube-api-access-ztfvd" pod "67ee4cb2-a701-44af-a9d3-538ef85c939c" (UID: "67ee4cb2-a701-44af-a9d3-538ef85c939c"). InnerVolumeSpecName "kube-api-access-ztfvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:19.577960 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.577925    4428 generic.go:356] "Generic (PLEG): container finished" podID="c622d692-1344-4482-9c4c-c26ae52a6642" containerID="8c5defb178a8e6c27c18b9978d94f82850c367db196518a2a2f102b132004c9e" exitCode=0
Oct 31 13:58:19.578109 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.577964    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96" event={"ID":"c622d692-1344-4482-9c4c-c26ae52a6642","Type":"ContainerDied","Data":"8c5defb178a8e6c27c18b9978d94f82850c367db196518a2a2f102b132004c9e"}
Oct 31 13:58:19.578109 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.577978    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"
Oct 31 13:58:19.578109 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.578028    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96" event={"ID":"c622d692-1344-4482-9c4c-c26ae52a6642","Type":"ContainerDied","Data":"b861e9570d13009858b9686f6d7bd3664ab88c2df542d84e6fb30f55de6932d6"}
Oct 31 13:58:19.578109 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.578060    4428 scope.go:122] "RemoveContainer" containerID="8c5defb178a8e6c27c18b9978d94f82850c367db196518a2a2f102b132004c9e"
Oct 31 13:58:19.580170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.580141    4428 generic.go:356] "Generic (PLEG): container finished" podID="67ee4cb2-a701-44af-a9d3-538ef85c939c" containerID="1413b9af9952455389618bcd762edca94c7c64246f3dae1aa8407cee48686dcb" exitCode=0
Oct 31 13:58:19.580315 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.580176    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"
Oct 31 13:58:19.580315 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.580214    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj" event={"ID":"67ee4cb2-a701-44af-a9d3-538ef85c939c","Type":"ContainerDied","Data":"1413b9af9952455389618bcd762edca94c7c64246f3dae1aa8407cee48686dcb"}
Oct 31 13:58:19.580315 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.580258    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj" event={"ID":"67ee4cb2-a701-44af-a9d3-538ef85c939c","Type":"ContainerDied","Data":"97e38df8bdb99f9cc4a743321a00ce9aadd910b5697e53d62f39bfb2bce208ce"}
Oct 31 13:58:19.593800 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.593780    4428 scope.go:122] "RemoveContainer" containerID="8c5defb178a8e6c27c18b9978d94f82850c367db196518a2a2f102b132004c9e"
Oct 31 13:58:19.594152 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:58:19.594125    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5defb178a8e6c27c18b9978d94f82850c367db196518a2a2f102b132004c9e\": container with ID starting with 8c5defb178a8e6c27c18b9978d94f82850c367db196518a2a2f102b132004c9e not found: ID does not exist" containerID="8c5defb178a8e6c27c18b9978d94f82850c367db196518a2a2f102b132004c9e"
Oct 31 13:58:19.594238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.594161    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5defb178a8e6c27c18b9978d94f82850c367db196518a2a2f102b132004c9e"} err="failed to get container status \"8c5defb178a8e6c27c18b9978d94f82850c367db196518a2a2f102b132004c9e\": rpc error: code = NotFound desc = could not find container \"8c5defb178a8e6c27c18b9978d94f82850c367db196518a2a2f102b132004c9e\": container with ID starting with 8c5defb178a8e6c27c18b9978d94f82850c367db196518a2a2f102b132004c9e not found: ID does not exist"
Oct 31 13:58:19.594238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.594187    4428 scope.go:122] "RemoveContainer" containerID="1413b9af9952455389618bcd762edca94c7c64246f3dae1aa8407cee48686dcb"
Oct 31 13:58:19.604683 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.604660    4428 scope.go:122] "RemoveContainer" containerID="1413b9af9952455389618bcd762edca94c7c64246f3dae1aa8407cee48686dcb"
Oct 31 13:58:19.605095 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:58:19.605061    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1413b9af9952455389618bcd762edca94c7c64246f3dae1aa8407cee48686dcb\": container with ID starting with 1413b9af9952455389618bcd762edca94c7c64246f3dae1aa8407cee48686dcb not found: ID does not exist" containerID="1413b9af9952455389618bcd762edca94c7c64246f3dae1aa8407cee48686dcb"
Oct 31 13:58:19.605194 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.605092    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1413b9af9952455389618bcd762edca94c7c64246f3dae1aa8407cee48686dcb"} err="failed to get container status \"1413b9af9952455389618bcd762edca94c7c64246f3dae1aa8407cee48686dcb\": rpc error: code = NotFound desc = could not find container \"1413b9af9952455389618bcd762edca94c7c64246f3dae1aa8407cee48686dcb\": container with ID starting with 1413b9af9952455389618bcd762edca94c7c64246f3dae1aa8407cee48686dcb not found: ID does not exist"
Oct 31 13:58:19.612262 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.612238    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"]
Oct 31 13:58:19.616730 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.616703    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6a3533-7fd5f5c8bc-2kwqj"]
Oct 31 13:58:19.625954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.625798    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"]
Oct 31 13:58:19.632162 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.632139    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6a3533-68fb85b995-ljs96"]
Oct 31 13:58:19.670325 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.670290    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztfvd\" (UniqueName: \"kubernetes.io/projected/67ee4cb2-a701-44af-a9d3-538ef85c939c-kube-api-access-ztfvd\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:20.009896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:19.997959    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:20.321958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:20.321196    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:20.487490 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:20.487454    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:20.548841 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:20.548805    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"]
Oct 31 13:58:20.549101 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:20.549067    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4" podUID="2db01e87-ab22-48aa-bed3-8b927d2a2162" containerName="istio-proxy" containerID="cri-o://e1cbccfe4abec796e5de1ddbb6f557171664543b4b882e604ce3965acb0d2c23" gracePeriod=30
Oct 31 13:58:20.685994 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:20.681326    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:20.854429 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:20.852110    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67ee4cb2-a701-44af-a9d3-538ef85c939c" path="/var/lib/kubelet/pods/67ee4cb2-a701-44af-a9d3-538ef85c939c/volumes"
Oct 31 13:58:20.854429 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:20.852823    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c622d692-1344-4482-9c4c-c26ae52a6642" path="/var/lib/kubelet/pods/c622d692-1344-4482-9c4c-c26ae52a6642/volumes"
Oct 31 13:58:21.180981 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:21.180939    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:22.189000 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:22.188710    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:23.017926 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:23.010758    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:23.379529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:23.379440    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:23.832208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:23.818468    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:24.618891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:24.611930    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:25.486138 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.481356    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:25.871607 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.871546    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:58:25.997289 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.997256    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2db01e87-ab22-48aa-bed3-8b927d2a2162-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db01e87-ab22-48aa-bed3-8b927d2a2162-istiod-ca-cert\") pod \"2db01e87-ab22-48aa-bed3-8b927d2a2162\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") "
Oct 31 13:58:25.997473 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.997377    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-data\") pod \"2db01e87-ab22-48aa-bed3-8b927d2a2162\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") "
Oct 31 13:58:25.997473 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.997458    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-token\" (UniqueName: \"kubernetes.io/projected/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-token\") pod \"2db01e87-ab22-48aa-bed3-8b927d2a2162\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") "
Oct 31 13:58:25.997592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.997491    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-credential-socket\") pod \"2db01e87-ab22-48aa-bed3-8b927d2a2162\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") "
Oct 31 13:58:25.997592 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.997524    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db01e87-ab22-48aa-bed3-8b927d2a2162-kube-api-access-bthx4\" (UniqueName: \"kubernetes.io/projected/2db01e87-ab22-48aa-bed3-8b927d2a2162-kube-api-access-bthx4\") pod \"2db01e87-ab22-48aa-bed3-8b927d2a2162\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") "
Oct 31 13:58:25.997707 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.997599    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-envoy\") pod \"2db01e87-ab22-48aa-bed3-8b927d2a2162\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") "
Oct 31 13:58:25.997707 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.997697    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-data" pod "2db01e87-ab22-48aa-bed3-8b927d2a2162" (UID: "2db01e87-ab22-48aa-bed3-8b927d2a2162"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:25.997818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.997730    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-workload-socket\") pod \"2db01e87-ab22-48aa-bed3-8b927d2a2162\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") "
Oct 31 13:58:25.997818 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.997783    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db01e87-ab22-48aa-bed3-8b927d2a2162-istiod-ca-cert" pod "2db01e87-ab22-48aa-bed3-8b927d2a2162" (UID: "2db01e87-ab22-48aa-bed3-8b927d2a2162"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:58:25.997968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.997829    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-workload-certs\") pod \"2db01e87-ab22-48aa-bed3-8b927d2a2162\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") "
Oct 31 13:58:25.997968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.997843    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-credential-socket" pod "2db01e87-ab22-48aa-bed3-8b927d2a2162" (UID: "2db01e87-ab22-48aa-bed3-8b927d2a2162"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:25.997968 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.997898    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-podinfo\") pod \"2db01e87-ab22-48aa-bed3-8b927d2a2162\" (UID: \"2db01e87-ab22-48aa-bed3-8b927d2a2162\") "
Oct 31 13:58:25.998124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.998043    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-workload-socket" pod "2db01e87-ab22-48aa-bed3-8b927d2a2162" (UID: "2db01e87-ab22-48aa-bed3-8b927d2a2162"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:25.998193 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.998168    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-workload-certs" pod "2db01e87-ab22-48aa-bed3-8b927d2a2162" (UID: "2db01e87-ab22-48aa-bed3-8b927d2a2162"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:25.998665 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.998636    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db01e87-ab22-48aa-bed3-8b927d2a2162-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:25.998665 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.998664    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:25.998907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.998675    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:25.998907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.998684    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:25.998907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:25.998693    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:26.000163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.000138    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-token" pod "2db01e87-ab22-48aa-bed3-8b927d2a2162" (UID: "2db01e87-ab22-48aa-bed3-8b927d2a2162"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:26.000285 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.000211    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-podinfo" pod "2db01e87-ab22-48aa-bed3-8b927d2a2162" (UID: "2db01e87-ab22-48aa-bed3-8b927d2a2162"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:58:26.000445 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.000421    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-envoy" pod "2db01e87-ab22-48aa-bed3-8b927d2a2162" (UID: "2db01e87-ab22-48aa-bed3-8b927d2a2162"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:26.000545 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.000460    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db01e87-ab22-48aa-bed3-8b927d2a2162-kube-api-access-bthx4" pod "2db01e87-ab22-48aa-bed3-8b927d2a2162" (UID: "2db01e87-ab22-48aa-bed3-8b927d2a2162"). InnerVolumeSpecName "kube-api-access-bthx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:26.099751 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.099658    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:26.099751 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.099692    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bthx4\" (UniqueName: \"kubernetes.io/projected/2db01e87-ab22-48aa-bed3-8b927d2a2162-kube-api-access-bthx4\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:26.099751 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.099707    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:26.099751 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.099719    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db01e87-ab22-48aa-bed3-8b927d2a2162-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:26.443233 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.443143    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:26.652018 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.651980    4428 generic.go:356] "Generic (PLEG): container finished" podID="2db01e87-ab22-48aa-bed3-8b927d2a2162" containerID="e1cbccfe4abec796e5de1ddbb6f557171664543b4b882e604ce3965acb0d2c23" exitCode=0
Oct 31 13:58:26.652245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.652025    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"
Oct 31 13:58:26.652245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.652046    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4" event={"ID":"2db01e87-ab22-48aa-bed3-8b927d2a2162","Type":"ContainerDied","Data":"e1cbccfe4abec796e5de1ddbb6f557171664543b4b882e604ce3965acb0d2c23"}
Oct 31 13:58:26.652245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.652129    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4" event={"ID":"2db01e87-ab22-48aa-bed3-8b927d2a2162","Type":"ContainerDied","Data":"9038216c9739971ac294698187d21a44665c08f2d3dfb981bc811b1a8c631f74"}
Oct 31 13:58:26.652245 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.652154    4428 scope.go:122] "RemoveContainer" containerID="e1cbccfe4abec796e5de1ddbb6f557171664543b4b882e604ce3965acb0d2c23"
Oct 31 13:58:26.666624 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.666598    4428 scope.go:122] "RemoveContainer" containerID="e1cbccfe4abec796e5de1ddbb6f557171664543b4b882e604ce3965acb0d2c23"
Oct 31 13:58:26.666965 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:58:26.666946    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1cbccfe4abec796e5de1ddbb6f557171664543b4b882e604ce3965acb0d2c23\": container with ID starting with e1cbccfe4abec796e5de1ddbb6f557171664543b4b882e604ce3965acb0d2c23 not found: ID does not exist" containerID="e1cbccfe4abec796e5de1ddbb6f557171664543b4b882e604ce3965acb0d2c23"
Oct 31 13:58:26.667027 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.666974    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1cbccfe4abec796e5de1ddbb6f557171664543b4b882e604ce3965acb0d2c23"} err="failed to get container status \"e1cbccfe4abec796e5de1ddbb6f557171664543b4b882e604ce3965acb0d2c23\": rpc error: code = NotFound desc = could not find container \"e1cbccfe4abec796e5de1ddbb6f557171664543b4b882e604ce3965acb0d2c23\": container with ID starting with e1cbccfe4abec796e5de1ddbb6f557171664543b4b882e604ce3965acb0d2c23 not found: ID does not exist"
Oct 31 13:58:26.680935 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.680907    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"]
Oct 31 13:58:26.685093 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.685059    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-xb9m4"]
Oct 31 13:58:26.758947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:26.758915    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2db01e87-ab22-48aa-bed3-8b927d2a2162" path="/var/lib/kubelet/pods/2db01e87-ab22-48aa-bed3-8b927d2a2162/volumes"
Oct 31 13:58:27.421970 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:27.421930    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:28.406902 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:28.406432    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:29.146437 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:29.146401    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:30.108834 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:30.108799    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:30.940912 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:30.940849    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:31.137096 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:31.137049    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"]
Oct 31 13:58:31.152756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:31.152719    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"]
Oct 31 13:58:31.284037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:31.284001    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:31.487070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:31.487010    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:31.599750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:31.599650    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"]
Oct 31 13:58:31.613775 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:31.613742    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"]
Oct 31 13:58:31.741086 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:31.741034    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:31.901850 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:31.901752    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:32.075893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:32.070038    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:32.242480 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:32.242444    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:33.195617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:33.195575    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:33.989605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:33.989516    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:34.719919 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:34.719471    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:35.829112 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:35.829077    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:36.138236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:36.138095    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr" podUID="a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f" containerName="payload-processing" containerID="cri-o://9d360ce6fc453349b02be282d92ec3febea3b2c78a263cf224f3205a2d47b455" gracePeriod=25
Oct 31 13:58:36.153737 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:36.153673    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996" podUID="ee72f133-ecfd-4910-8026-e0da592ead60" containerName="payload-pre-processing" containerID="cri-o://ca30dec0fb80e96c9a4e758e406b69526a568c2e7cf94479b69f1566b60e81c6" gracePeriod=25
Oct 31 13:58:36.600861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:36.600816    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4" podUID="d5d381df-edd5-406d-b348-1d0ffc55b612" containerName="payload-processing" containerID="cri-o://705d3e8ef999eddd1ba32588b540f84ba6bf3dc124297db3a09aac68fcb3a257" gracePeriod=25
Oct 31 13:58:36.615406 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:36.615367    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689" podUID="d4ad25ff-2a15-410b-8bb3-96633053d27d" containerName="payload-pre-processing" containerID="cri-o://2873947a16a03403208bdc29f3ac6f8931158ce8d913ce50fc8c9058e9c41064" gracePeriod=25
Oct 31 13:58:36.743188 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:36.743160    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:36.771784 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:36.771738    4428 generic.go:356] "Generic (PLEG): container finished" podID="d4ad25ff-2a15-410b-8bb3-96633053d27d" containerID="2873947a16a03403208bdc29f3ac6f8931158ce8d913ce50fc8c9058e9c41064" exitCode=0
Oct 31 13:58:36.772161 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:36.772049    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689" event={"ID":"d4ad25ff-2a15-410b-8bb3-96633053d27d","Type":"ContainerDied","Data":"2873947a16a03403208bdc29f3ac6f8931158ce8d913ce50fc8c9058e9c41064"}
Oct 31 13:58:36.774205 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:36.774180    4428 generic.go:356] "Generic (PLEG): container finished" podID="d5d381df-edd5-406d-b348-1d0ffc55b612" containerID="705d3e8ef999eddd1ba32588b540f84ba6bf3dc124297db3a09aac68fcb3a257" exitCode=0
Oct 31 13:58:36.774346 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:36.774318    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4" event={"ID":"d5d381df-edd5-406d-b348-1d0ffc55b612","Type":"ContainerDied","Data":"705d3e8ef999eddd1ba32588b540f84ba6bf3dc124297db3a09aac68fcb3a257"}
Oct 31 13:58:36.932067 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:36.932027    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"
Oct 31 13:58:36.983236 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:36.983208    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"
Oct 31 13:58:37.035995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.035955    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d381df-edd5-406d-b348-1d0ffc55b612-kube-api-access-p9n2f\" (UniqueName: \"kubernetes.io/projected/d5d381df-edd5-406d-b348-1d0ffc55b612-kube-api-access-p9n2f\") pod \"d5d381df-edd5-406d-b348-1d0ffc55b612\" (UID: \"d5d381df-edd5-406d-b348-1d0ffc55b612\") "
Oct 31 13:58:37.036207 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.036175    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d381df-edd5-406d-b348-1d0ffc55b612-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d381df-edd5-406d-b348-1d0ffc55b612-plugins-config-volume\") pod \"d5d381df-edd5-406d-b348-1d0ffc55b612\" (UID: \"d5d381df-edd5-406d-b348-1d0ffc55b612\") "
Oct 31 13:58:37.036604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.036545    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d381df-edd5-406d-b348-1d0ffc55b612-plugins-config-volume" pod "d5d381df-edd5-406d-b348-1d0ffc55b612" (UID: "d5d381df-edd5-406d-b348-1d0ffc55b612"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:58:37.036889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.036853    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d381df-edd5-406d-b348-1d0ffc55b612-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:37.038648 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.038614    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d381df-edd5-406d-b348-1d0ffc55b612-kube-api-access-p9n2f" pod "d5d381df-edd5-406d-b348-1d0ffc55b612" (UID: "d5d381df-edd5-406d-b348-1d0ffc55b612"). InnerVolumeSpecName "kube-api-access-p9n2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:37.111456 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.111369    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"]
Oct 31 13:58:37.138383 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.138350    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4ad25ff-2a15-410b-8bb3-96633053d27d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ad25ff-2a15-410b-8bb3-96633053d27d-plugins-config-volume\") pod \"d4ad25ff-2a15-410b-8bb3-96633053d27d\" (UID: \"d4ad25ff-2a15-410b-8bb3-96633053d27d\") "
Oct 31 13:58:37.138909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.138834    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ad25ff-2a15-410b-8bb3-96633053d27d-kube-api-access-fhh2m\" (UniqueName: \"kubernetes.io/projected/d4ad25ff-2a15-410b-8bb3-96633053d27d-kube-api-access-fhh2m\") pod \"d4ad25ff-2a15-410b-8bb3-96633053d27d\" (UID: \"d4ad25ff-2a15-410b-8bb3-96633053d27d\") "
Oct 31 13:58:37.139617 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.139592    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9n2f\" (UniqueName: \"kubernetes.io/projected/d5d381df-edd5-406d-b348-1d0ffc55b612-kube-api-access-p9n2f\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:37.142976 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.142941    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ad25ff-2a15-410b-8bb3-96633053d27d-plugins-config-volume" pod "d4ad25ff-2a15-410b-8bb3-96633053d27d" (UID: "d4ad25ff-2a15-410b-8bb3-96633053d27d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:58:37.143182 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.143134    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ad25ff-2a15-410b-8bb3-96633053d27d-kube-api-access-fhh2m" pod "d4ad25ff-2a15-410b-8bb3-96633053d27d" (UID: "d4ad25ff-2a15-410b-8bb3-96633053d27d"). InnerVolumeSpecName "kube-api-access-fhh2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:37.237828 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.237775    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:37.241613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.241586    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhh2m\" (UniqueName: \"kubernetes.io/projected/d4ad25ff-2a15-410b-8bb3-96633053d27d-kube-api-access-fhh2m\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:37.241613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.241612    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ad25ff-2a15-410b-8bb3-96633053d27d-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:37.401951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.398697    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:37.580965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.580861    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:37.784137 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.784105    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"
Oct 31 13:58:37.784137 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.784122    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689" event={"ID":"d4ad25ff-2a15-410b-8bb3-96633053d27d","Type":"ContainerDied","Data":"8736a42d11980bd995a3bc09785678ddc9a198024862c66a371ba34cb6eb3aa4"}
Oct 31 13:58:37.784435 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.784191    4428 scope.go:122] "RemoveContainer" containerID="2873947a16a03403208bdc29f3ac6f8931158ce8d913ce50fc8c9058e9c41064"
Oct 31 13:58:37.785830 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.785814    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"
Oct 31 13:58:37.786019 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.785995    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4" event={"ID":"d5d381df-edd5-406d-b348-1d0ffc55b612","Type":"ContainerDied","Data":"1f128dc08bfea22c157aa5ba3e83702816225ba2a72ea2705519d4e486cc5950"}
Oct 31 13:58:37.797300 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.797274    4428 scope.go:122] "RemoveContainer" containerID="705d3e8ef999eddd1ba32588b540f84ba6bf3dc124297db3a09aac68fcb3a257"
Oct 31 13:58:37.827927 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.827857    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"]
Oct 31 13:58:37.833854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.833823    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-85549bb9f-hk689"]
Oct 31 13:58:37.844720 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.844690    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"]
Oct 31 13:58:37.851072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.851044    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f9103-5bd959fd85-t6xg4"]
Oct 31 13:58:37.923189 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:37.923156    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:38.641089 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:38.641039    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:38.759527 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:38.759494    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ad25ff-2a15-410b-8bb3-96633053d27d" path="/var/lib/kubelet/pods/d4ad25ff-2a15-410b-8bb3-96633053d27d/volumes"
Oct 31 13:58:38.760048 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:38.760032    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d381df-edd5-406d-b348-1d0ffc55b612" path="/var/lib/kubelet/pods/d5d381df-edd5-406d-b348-1d0ffc55b612/volumes"
Oct 31 13:58:39.428893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:39.423966    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:39.937581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:39.931424    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"]
Oct 31 13:58:39.937581 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:39.931702    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7" podUID="35b067ff-bd37-4cc5-85c9-d877a435751e" containerName="maas-api" containerID="cri-o://4109e0bd0edf309d1628e404b29a16eb685a6f3ac2dba9e030ff07a0af6a27f6" gracePeriod=30
Oct 31 13:58:40.033742 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.033697    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:40.231893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.227396    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:40.393494 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.393459    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:40.446529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.446501    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:58:40.488360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.488280    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35b067ff-bd37-4cc5-85c9-d877a435751e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b067ff-bd37-4cc5-85c9-d877a435751e-metrics-certs\") pod \"35b067ff-bd37-4cc5-85c9-d877a435751e\" (UID: \"35b067ff-bd37-4cc5-85c9-d877a435751e\") "
Oct 31 13:58:40.488360 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.488324    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35b067ff-bd37-4cc5-85c9-d877a435751e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35b067ff-bd37-4cc5-85c9-d877a435751e-maas-api-tls\") pod \"35b067ff-bd37-4cc5-85c9-d877a435751e\" (UID: \"35b067ff-bd37-4cc5-85c9-d877a435751e\") "
Oct 31 13:58:40.488612 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.488535    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b067ff-bd37-4cc5-85c9-d877a435751e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b067ff-bd37-4cc5-85c9-d877a435751e-openshift-service-ca\") pod \"35b067ff-bd37-4cc5-85c9-d877a435751e\" (UID: \"35b067ff-bd37-4cc5-85c9-d877a435751e\") "
Oct 31 13:58:40.488612 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.488579    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b067ff-bd37-4cc5-85c9-d877a435751e-kube-api-access-l7rhl\" (UniqueName: \"kubernetes.io/projected/35b067ff-bd37-4cc5-85c9-d877a435751e-kube-api-access-l7rhl\") pod \"35b067ff-bd37-4cc5-85c9-d877a435751e\" (UID: \"35b067ff-bd37-4cc5-85c9-d877a435751e\") "
Oct 31 13:58:40.488954 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.488926    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b067ff-bd37-4cc5-85c9-d877a435751e-openshift-service-ca" pod "35b067ff-bd37-4cc5-85c9-d877a435751e" (UID: "35b067ff-bd37-4cc5-85c9-d877a435751e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:58:40.489259 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.489204    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b067ff-bd37-4cc5-85c9-d877a435751e-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:40.490764 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.490738    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b067ff-bd37-4cc5-85c9-d877a435751e-metrics-certs" pod "35b067ff-bd37-4cc5-85c9-d877a435751e" (UID: "35b067ff-bd37-4cc5-85c9-d877a435751e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:58:40.491211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.491185    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b067ff-bd37-4cc5-85c9-d877a435751e-kube-api-access-l7rhl" pod "35b067ff-bd37-4cc5-85c9-d877a435751e" (UID: "35b067ff-bd37-4cc5-85c9-d877a435751e"). InnerVolumeSpecName "kube-api-access-l7rhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:40.491359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.491339    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b067ff-bd37-4cc5-85c9-d877a435751e-maas-api-tls" pod "35b067ff-bd37-4cc5-85c9-d877a435751e" (UID: "35b067ff-bd37-4cc5-85c9-d877a435751e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:58:40.590576 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.590536    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b067ff-bd37-4cc5-85c9-d877a435751e-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:40.590576 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.590569    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35b067ff-bd37-4cc5-85c9-d877a435751e-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:40.590576 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.590579    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7rhl\" (UniqueName: \"kubernetes.io/projected/35b067ff-bd37-4cc5-85c9-d877a435751e-kube-api-access-l7rhl\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:40.710908 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.700798    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:40.813687 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.813647    4428 generic.go:356] "Generic (PLEG): container finished" podID="35b067ff-bd37-4cc5-85c9-d877a435751e" containerID="4109e0bd0edf309d1628e404b29a16eb685a6f3ac2dba9e030ff07a0af6a27f6" exitCode=0
Oct 31 13:58:40.813924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.813703    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"
Oct 31 13:58:40.813924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.813724    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7" event={"ID":"35b067ff-bd37-4cc5-85c9-d877a435751e","Type":"ContainerDied","Data":"4109e0bd0edf309d1628e404b29a16eb685a6f3ac2dba9e030ff07a0af6a27f6"}
Oct 31 13:58:40.813924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.813782    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7" event={"ID":"35b067ff-bd37-4cc5-85c9-d877a435751e","Type":"ContainerDied","Data":"c2c7072230bc7877a9ffb11f4da1dce2e8d2e19c21245ccac88c0e3f3a4e010c"}
Oct 31 13:58:40.813924 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.813812    4428 scope.go:122] "RemoveContainer" containerID="4109e0bd0edf309d1628e404b29a16eb685a6f3ac2dba9e030ff07a0af6a27f6"
Oct 31 13:58:40.824666 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.824646    4428 scope.go:122] "RemoveContainer" containerID="4109e0bd0edf309d1628e404b29a16eb685a6f3ac2dba9e030ff07a0af6a27f6"
Oct 31 13:58:40.824979 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:58:40.824955    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4109e0bd0edf309d1628e404b29a16eb685a6f3ac2dba9e030ff07a0af6a27f6\": container with ID starting with 4109e0bd0edf309d1628e404b29a16eb685a6f3ac2dba9e030ff07a0af6a27f6 not found: ID does not exist" containerID="4109e0bd0edf309d1628e404b29a16eb685a6f3ac2dba9e030ff07a0af6a27f6"
Oct 31 13:58:40.825064 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.824989    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4109e0bd0edf309d1628e404b29a16eb685a6f3ac2dba9e030ff07a0af6a27f6"} err="failed to get container status \"4109e0bd0edf309d1628e404b29a16eb685a6f3ac2dba9e030ff07a0af6a27f6\": rpc error: code = NotFound desc = could not find container \"4109e0bd0edf309d1628e404b29a16eb685a6f3ac2dba9e030ff07a0af6a27f6\": container with ID starting with 4109e0bd0edf309d1628e404b29a16eb685a6f3ac2dba9e030ff07a0af6a27f6 not found: ID does not exist"
Oct 31 13:58:40.841347 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.841318    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"]
Oct 31 13:58:40.845492 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.845465    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-612b46-55f5558c86-hzbj7"]
Oct 31 13:58:40.970718 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:40.970682    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:41.626161 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:41.626122    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:42.112231 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.112189    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2" podUID="e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0" containerName="payload-pre-processing" containerID="cri-o://583105638b068265a65e08653802d465ecc11e1c2684e4598bf1a4076ddd0c2e" gracePeriod=25
Oct 31 13:58:42.327847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.326677    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:42.467916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.467860    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"
Oct 31 13:58:42.528992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.528953    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0-kube-api-access-kq4q7\" (UniqueName: \"kubernetes.io/projected/e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0-kube-api-access-kq4q7\") pod \"e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0\" (UID: \"e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0\") "
Oct 31 13:58:42.529228 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.529211    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0-plugins-config-volume\") pod \"e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0\" (UID: \"e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0\") "
Oct 31 13:58:42.529649 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.529618    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0-plugins-config-volume" pod "e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0" (UID: "e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:58:42.530097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.530075    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:42.531602 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.531572    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0-kube-api-access-kq4q7" pod "e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0" (UID: "e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0"). InnerVolumeSpecName "kube-api-access-kq4q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:42.631438 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.631397    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq4q7\" (UniqueName: \"kubernetes.io/projected/e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0-kube-api-access-kq4q7\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:42.757719 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.757686    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35b067ff-bd37-4cc5-85c9-d877a435751e" path="/var/lib/kubelet/pods/35b067ff-bd37-4cc5-85c9-d877a435751e/volumes"
Oct 31 13:58:42.835574 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.835169    4428 generic.go:356] "Generic (PLEG): container finished" podID="e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0" containerID="583105638b068265a65e08653802d465ecc11e1c2684e4598bf1a4076ddd0c2e" exitCode=0
Oct 31 13:58:42.835574 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.835208    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2" event={"ID":"e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0","Type":"ContainerDied","Data":"583105638b068265a65e08653802d465ecc11e1c2684e4598bf1a4076ddd0c2e"}
Oct 31 13:58:42.835574 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.835246    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"
Oct 31 13:58:42.835574 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.835286    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2" event={"ID":"e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0","Type":"ContainerDied","Data":"7ba908e2b8afd3b4c55e42f9d44fb15fa74e25e17048b92a2026ebb145e1dcea"}
Oct 31 13:58:42.835574 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.835316    4428 scope.go:122] "RemoveContainer" containerID="583105638b068265a65e08653802d465ecc11e1c2684e4598bf1a4076ddd0c2e"
Oct 31 13:58:42.848718 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.848690    4428 scope.go:122] "RemoveContainer" containerID="583105638b068265a65e08653802d465ecc11e1c2684e4598bf1a4076ddd0c2e"
Oct 31 13:58:42.849084 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:58:42.849062    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"583105638b068265a65e08653802d465ecc11e1c2684e4598bf1a4076ddd0c2e\": container with ID starting with 583105638b068265a65e08653802d465ecc11e1c2684e4598bf1a4076ddd0c2e not found: ID does not exist" containerID="583105638b068265a65e08653802d465ecc11e1c2684e4598bf1a4076ddd0c2e"
Oct 31 13:58:42.849186 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.849098    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"583105638b068265a65e08653802d465ecc11e1c2684e4598bf1a4076ddd0c2e"} err="failed to get container status \"583105638b068265a65e08653802d465ecc11e1c2684e4598bf1a4076ddd0c2e\": rpc error: code = NotFound desc = could not find container \"583105638b068265a65e08653802d465ecc11e1c2684e4598bf1a4076ddd0c2e\": container with ID starting with 583105638b068265a65e08653802d465ecc11e1c2684e4598bf1a4076ddd0c2e not found: ID does not exist"
Oct 31 13:58:42.859652 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.859620    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"]
Oct 31 13:58:42.864693 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:42.864666    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-65b55fd7f-mdvb2"]
Oct 31 13:58:43.298672 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:43.298629    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:43.453980 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:43.453943    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:44.185505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:44.185471    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:44.758543 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:44.758510    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0" path="/var/lib/kubelet/pods/e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0/volumes"
Oct 31 13:58:44.926893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:44.921366    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:45.423935 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:45.423861    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996" podUID="ee72f133-ecfd-4910-8026-e0da592ead60" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:58:45.424179 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:45.424155    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr" podUID="a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:58:45.815911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:45.802717    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:46.825063 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:46.825021    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:47.478455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.478419    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:47.618321 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.618284    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"]
Oct 31 13:58:47.619072 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619057    4428 cpu_manager.go:414] "containerMap: removing container" podUID="c622d692-1344-4482-9c4c-c26ae52a6642" containerName="payload-processing"
Oct 31 13:58:47.619133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619073    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c622d692-1344-4482-9c4c-c26ae52a6642" containerName="payload-processing"
Oct 31 13:58:47.619133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619089    4428 cpu_manager.go:414] "containerMap: removing container" podUID="2db01e87-ab22-48aa-bed3-8b927d2a2162" containerName="istio-proxy"
Oct 31 13:58:47.619133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619094    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db01e87-ab22-48aa-bed3-8b927d2a2162" containerName="istio-proxy"
Oct 31 13:58:47.619133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619106    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0" containerName="payload-pre-processing"
Oct 31 13:58:47.619133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619111    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0" containerName="payload-pre-processing"
Oct 31 13:58:47.619133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619119    4428 cpu_manager.go:414] "containerMap: removing container" podUID="67ee4cb2-a701-44af-a9d3-538ef85c939c" containerName="payload-pre-processing"
Oct 31 13:58:47.619133 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619124    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ee4cb2-a701-44af-a9d3-538ef85c939c" containerName="payload-pre-processing"
Oct 31 13:58:47.619370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619141    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b844988a-dac6-463f-898b-ec37e3447ff1" containerName="maas-api"
Oct 31 13:58:47.619370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619146    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b844988a-dac6-463f-898b-ec37e3447ff1" containerName="maas-api"
Oct 31 13:58:47.619370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619153    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d5d381df-edd5-406d-b348-1d0ffc55b612" containerName="payload-processing"
Oct 31 13:58:47.619370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619157    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d381df-edd5-406d-b348-1d0ffc55b612" containerName="payload-processing"
Oct 31 13:58:47.619370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619165    4428 cpu_manager.go:414] "containerMap: removing container" podUID="35b067ff-bd37-4cc5-85c9-d877a435751e" containerName="maas-api"
Oct 31 13:58:47.619370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619169    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b067ff-bd37-4cc5-85c9-d877a435751e" containerName="maas-api"
Oct 31 13:58:47.619370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619191    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d4ad25ff-2a15-410b-8bb3-96633053d27d" containerName="payload-pre-processing"
Oct 31 13:58:47.619370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619195    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ad25ff-2a15-410b-8bb3-96633053d27d" containerName="payload-pre-processing"
Oct 31 13:58:47.619370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619305    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db01e87-ab22-48aa-bed3-8b927d2a2162" containerName="istio-proxy"
Oct 31 13:58:47.619370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619314    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3aa47ef-57eb-4f88-ba2c-d07f9ed188c0" containerName="payload-pre-processing"
Oct 31 13:58:47.619370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619320    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ee4cb2-a701-44af-a9d3-538ef85c939c" containerName="payload-pre-processing"
Oct 31 13:58:47.619370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619327    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b844988a-dac6-463f-898b-ec37e3447ff1" containerName="maas-api"
Oct 31 13:58:47.619370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619335    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d381df-edd5-406d-b348-1d0ffc55b612" containerName="payload-processing"
Oct 31 13:58:47.619370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619340    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b067ff-bd37-4cc5-85c9-d877a435751e" containerName="maas-api"
Oct 31 13:58:47.619370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619348    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ad25ff-2a15-410b-8bb3-96633053d27d" containerName="payload-pre-processing"
Oct 31 13:58:47.619370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.619358    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c622d692-1344-4482-9c4c-c26ae52a6642" containerName="payload-processing"
Oct 31 13:58:47.624579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.624555    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"
Oct 31 13:58:47.636621 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.635062    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"]
Oct 31 13:58:47.636938 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.636914    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9f9103\""
Oct 31 13:58:47.637100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.637045    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9f9103-dockercfg-b6cn5\""
Oct 31 13:58:47.656310 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.656279    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:47.753947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.742571    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb433fc-0251-4148-b9b3-ec57bf90be3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d\" (UID: \"fdb433fc-0251-4148-b9b3-ec57bf90be3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"
Oct 31 13:58:47.753947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.742943    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rcqj\" (UniqueName: \"kubernetes.io/projected/fdb433fc-0251-4148-b9b3-ec57bf90be3a-kube-api-access-4rcqj\") pod \"payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d\" (UID: \"fdb433fc-0251-4148-b9b3-ec57bf90be3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"
Oct 31 13:58:47.824898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.820931    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"]
Oct 31 13:58:47.867893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.867689    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"
Oct 31 13:58:47.883893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.882246    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rcqj\" (UniqueName: \"kubernetes.io/projected/fdb433fc-0251-4148-b9b3-ec57bf90be3a-kube-api-access-4rcqj\") pod \"payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d\" (UID: \"fdb433fc-0251-4148-b9b3-ec57bf90be3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"
Oct 31 13:58:47.883893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.883177    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb433fc-0251-4148-b9b3-ec57bf90be3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d\" (UID: \"fdb433fc-0251-4148-b9b3-ec57bf90be3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"
Oct 31 13:58:47.887943 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.884353    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb433fc-0251-4148-b9b3-ec57bf90be3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d\" (UID: \"fdb433fc-0251-4148-b9b3-ec57bf90be3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"
Oct 31 13:58:47.937170 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.937136    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rcqj\" (UniqueName: \"kubernetes.io/projected/fdb433fc-0251-4148-b9b3-ec57bf90be3a-kube-api-access-4rcqj\") pod \"payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d\" (UID: \"fdb433fc-0251-4148-b9b3-ec57bf90be3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"
Oct 31 13:58:47.946840 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.946803    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"
Oct 31 13:58:47.956790 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.956756    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"]
Oct 31 13:58:47.984518 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.984480    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf34172e-3804-4277-aeab-900beeaeefe7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96\" (UID: \"cf34172e-3804-4277-aeab-900beeaeefe7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"
Oct 31 13:58:47.984854 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:47.984823    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j79w9\" (UniqueName: \"kubernetes.io/projected/cf34172e-3804-4277-aeab-900beeaeefe7-kube-api-access-j79w9\") pod \"payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96\" (UID: \"cf34172e-3804-4277-aeab-900beeaeefe7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"
Oct 31 13:58:48.086426 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:48.086386    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j79w9\" (UniqueName: \"kubernetes.io/projected/cf34172e-3804-4277-aeab-900beeaeefe7-kube-api-access-j79w9\") pod \"payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96\" (UID: \"cf34172e-3804-4277-aeab-900beeaeefe7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"
Oct 31 13:58:48.086635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:48.086507    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf34172e-3804-4277-aeab-900beeaeefe7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96\" (UID: \"cf34172e-3804-4277-aeab-900beeaeefe7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"
Oct 31 13:58:48.087239 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:48.087216    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf34172e-3804-4277-aeab-900beeaeefe7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96\" (UID: \"cf34172e-3804-4277-aeab-900beeaeefe7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"
Oct 31 13:58:48.100200 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:48.100169    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j79w9\" (UniqueName: \"kubernetes.io/projected/cf34172e-3804-4277-aeab-900beeaeefe7-kube-api-access-j79w9\") pod \"payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96\" (UID: \"cf34172e-3804-4277-aeab-900beeaeefe7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"
Oct 31 13:58:48.241080 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:48.241042    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"
Oct 31 13:58:48.248646 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:58:48.248598    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb433fc_0251_4148_b9b3_ec57bf90be3a.slice/crio-81397089ee03c1a571a372137cedf14a1172b06be10503e40e1a0e47793c0d5c WatchSource:0}: Error finding container 81397089ee03c1a571a372137cedf14a1172b06be10503e40e1a0e47793c0d5c: Status 404 returned error can't find the container with id 81397089ee03c1a571a372137cedf14a1172b06be10503e40e1a0e47793c0d5c
Oct 31 13:58:48.249817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:48.249788    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"]
Oct 31 13:58:48.328639 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:48.324300    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:48.617100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:48.617072    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"]
Oct 31 13:58:48.617577 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:58:48.617551    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf34172e_3804_4277_aeab_900beeaeefe7.slice/crio-3a51eeb86fcda9e63f9fdcc56f4a83eab7cecde9b6c9e9159d1f2aede58103df WatchSource:0}: Error finding container 3a51eeb86fcda9e63f9fdcc56f4a83eab7cecde9b6c9e9159d1f2aede58103df: Status 404 returned error can't find the container with id 3a51eeb86fcda9e63f9fdcc56f4a83eab7cecde9b6c9e9159d1f2aede58103df
Oct 31 13:58:48.741938 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:58:48.741898    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:58:48.957807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:48.957496    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d" event={"ID":"fdb433fc-0251-4148-b9b3-ec57bf90be3a","Type":"ContainerStarted","Data":"a50763f471afb7baa9cbf848e97357119d92b7c6375175301e37e668ee0a6dff"}
Oct 31 13:58:48.957807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:48.957590    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d" event={"ID":"fdb433fc-0251-4148-b9b3-ec57bf90be3a","Type":"ContainerStarted","Data":"81397089ee03c1a571a372137cedf14a1172b06be10503e40e1a0e47793c0d5c"}
Oct 31 13:58:48.957807 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:48.957659    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"
Oct 31 13:58:48.959784 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:48.959752    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96" event={"ID":"cf34172e-3804-4277-aeab-900beeaeefe7","Type":"ContainerStarted","Data":"4ab0e5eb28a04084a4da3041e0a04ff0888a889f8126fdba35d74d1f955ee9bc"}
Oct 31 13:58:48.959985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:48.959807    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96" event={"ID":"cf34172e-3804-4277-aeab-900beeaeefe7","Type":"ContainerStarted","Data":"3a51eeb86fcda9e63f9fdcc56f4a83eab7cecde9b6c9e9159d1f2aede58103df"}
Oct 31 13:58:48.959985 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:48.959839    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"
Oct 31 13:58:48.988606 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:48.988540    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d" podStartSLOduration=1.988519008 podStartE2EDuration="1.988519008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:58:48.983433106 +0000 UTC m=+3326.381781732" watchObservedRunningTime="2026-10-31 13:58:48.988519008 +0000 UTC m=+3326.386867634"
Oct 31 13:58:49.021350 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:49.021288    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96" podStartSLOduration=2.021269654 podStartE2EDuration="2.021269654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:58:49.020202389 +0000 UTC m=+3326.418551012" watchObservedRunningTime="2026-10-31 13:58:49.021269654 +0000 UTC m=+3326.419618280"
Oct 31 13:58:49.258891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:49.252488    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:49.638958 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:49.632924    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:49.792560 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:49.792522    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:49.893011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:49.892349    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"]
Oct 31 13:58:49.893011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:49.892630    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb" podUID="fdbe827a-0f8e-41f0-a10e-0893d404efbe" containerName="istio-proxy" containerID="cri-o://ad32ac23a1e52aad0193f0a3066df81caf88785204b17ba2243c13ff8f27bda2" gracePeriod=30
Oct 31 13:58:49.987891 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:49.986994    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:50.325889 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:50.318899    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:50.492472 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:50.492438    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:51.635922 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:51.635135    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:52.395142 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:52.395096    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:53.130897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:53.130066    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:53.882945 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:53.873237    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:54.672028 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:54.671990    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:55.099017 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.098981    4428 generic.go:356] "Generic (PLEG): container finished" podID="fdbe827a-0f8e-41f0-a10e-0893d404efbe" containerID="ad32ac23a1e52aad0193f0a3066df81caf88785204b17ba2243c13ff8f27bda2" exitCode=0
Oct 31 13:58:55.099187 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.099063    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb" event={"ID":"fdbe827a-0f8e-41f0-a10e-0893d404efbe","Type":"ContainerDied","Data":"ad32ac23a1e52aad0193f0a3066df81caf88785204b17ba2243c13ff8f27bda2"}
Oct 31 13:58:55.099187 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.099149    4428 scope.go:122] "RemoveContainer" containerID="e6ee5852d7a7f2ef0706c0175847e8eb8744833445e552a8018fd670b0eec565"
Oct 31 13:58:55.251642 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.251615    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:58:55.295601 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.295562    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-credential-socket\") pod \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") "
Oct 31 13:58:55.295810 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.295738    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdbe827a-0f8e-41f0-a10e-0893d404efbe-kube-api-access-4gtms\" (UniqueName: \"kubernetes.io/projected/fdbe827a-0f8e-41f0-a10e-0893d404efbe-kube-api-access-4gtms\") pod \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") "
Oct 31 13:58:55.295929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.295823    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-podinfo\") pod \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") "
Oct 31 13:58:55.295929 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.295900    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istiod-ca-cert\") pod \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") "
Oct 31 13:58:55.296088 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.295919    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-credential-socket" pod "fdbe827a-0f8e-41f0-a10e-0893d404efbe" (UID: "fdbe827a-0f8e-41f0-a10e-0893d404efbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:55.296088 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.295992    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-data\") pod \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") "
Oct 31 13:58:55.296199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.296089    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-token\" (UniqueName: \"kubernetes.io/projected/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-token\") pod \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") "
Oct 31 13:58:55.296199 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.296162    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-envoy\") pod \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") "
Oct 31 13:58:55.296317 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.296205    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-workload-certs\") pod \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") "
Oct 31 13:58:55.296317 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.296259    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-workload-socket\") pod \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\" (UID: \"fdbe827a-0f8e-41f0-a10e-0893d404efbe\") "
Oct 31 13:58:55.296940 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.296637    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-data" pod "fdbe827a-0f8e-41f0-a10e-0893d404efbe" (UID: "fdbe827a-0f8e-41f0-a10e-0893d404efbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:55.296940 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.296904    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-workload-socket" pod "fdbe827a-0f8e-41f0-a10e-0893d404efbe" (UID: "fdbe827a-0f8e-41f0-a10e-0893d404efbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:55.297750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.297426    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:55.297750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.297456    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:55.297750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.297471    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:55.300238 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.300213    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbe827a-0f8e-41f0-a10e-0893d404efbe-kube-api-access-4gtms" pod "fdbe827a-0f8e-41f0-a10e-0893d404efbe" (UID: "fdbe827a-0f8e-41f0-a10e-0893d404efbe"). InnerVolumeSpecName "kube-api-access-4gtms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:55.301359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.301330    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-podinfo" pod "fdbe827a-0f8e-41f0-a10e-0893d404efbe" (UID: "fdbe827a-0f8e-41f0-a10e-0893d404efbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:58:55.301359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.301333    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-token" pod "fdbe827a-0f8e-41f0-a10e-0893d404efbe" (UID: "fdbe827a-0f8e-41f0-a10e-0893d404efbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:55.301579 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.301555    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-workload-certs" pod "fdbe827a-0f8e-41f0-a10e-0893d404efbe" (UID: "fdbe827a-0f8e-41f0-a10e-0893d404efbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:55.301769 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.301745    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istiod-ca-cert" pod "fdbe827a-0f8e-41f0-a10e-0893d404efbe" (UID: "fdbe827a-0f8e-41f0-a10e-0893d404efbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:58:55.302789 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.302766    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-envoy" pod "fdbe827a-0f8e-41f0-a10e-0893d404efbe" (UID: "fdbe827a-0f8e-41f0-a10e-0893d404efbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:58:55.365636 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.365602    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"]
Oct 31 13:58:55.400213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.400071    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:55.400213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.400127    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:55.400213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.400142    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbe827a-0f8e-41f0-a10e-0893d404efbe-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:55.400213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.400156    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gtms\" (UniqueName: \"kubernetes.io/projected/fdbe827a-0f8e-41f0-a10e-0893d404efbe-kube-api-access-4gtms\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:55.400213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.400189    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:55.400213 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.400203    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbe827a-0f8e-41f0-a10e-0893d404efbe-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:55.423317 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.423277    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996" podUID="ee72f133-ecfd-4910-8026-e0da592ead60" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:58:55.423505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.423443    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr" podUID="a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:58:55.502666 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.502628    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:55.661374 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:55.661339    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:56.108409 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.108381    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"
Oct 31 13:58:56.108950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.108416    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb" event={"ID":"fdbe827a-0f8e-41f0-a10e-0893d404efbe","Type":"ContainerDied","Data":"fa705bbd3889f36066eba8aec4290966045e15f05d037f8599682234e1ebfdba"}
Oct 31 13:58:56.108950 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.108490    4428 scope.go:122] "RemoveContainer" containerID="ad32ac23a1e52aad0193f0a3066df81caf88785204b17ba2243c13ff8f27bda2"
Oct 31 13:58:56.159267 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.159231    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"]
Oct 31 13:58:56.165345 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.164532    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-612b46-openshift-default-79fc6676c5-b7mfb"]
Oct 31 13:58:56.170319 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.170292    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:56.287143 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.287103    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"]
Oct 31 13:58:56.287987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.287965    4428 cpu_manager.go:414] "containerMap: removing container" podUID="fdbe827a-0f8e-41f0-a10e-0893d404efbe" containerName="istio-proxy"
Oct 31 13:58:56.287987 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.287985    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdbe827a-0f8e-41f0-a10e-0893d404efbe" containerName="istio-proxy"
Oct 31 13:58:56.288208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.288022    4428 cpu_manager.go:414] "containerMap: removing container" podUID="fdbe827a-0f8e-41f0-a10e-0893d404efbe" containerName="istio-proxy"
Oct 31 13:58:56.288208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.288028    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdbe827a-0f8e-41f0-a10e-0893d404efbe" containerName="istio-proxy"
Oct 31 13:58:56.288208 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.288169    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdbe827a-0f8e-41f0-a10e-0893d404efbe" containerName="istio-proxy"
Oct 31 13:58:56.292130 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.292106    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"
Oct 31 13:58:56.304147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.300970    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5af6eb-dockercfg-fxt4h\""
Oct 31 13:58:56.304147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.301075    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5af6eb\""
Oct 31 13:58:56.304147 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.304027    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"]
Oct 31 13:58:56.315324 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.313964    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacfb744-b683-4b4d-a4e8-eefeaa418d71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt\" (UID: \"bacfb744-b683-4b4d-a4e8-eefeaa418d71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"
Oct 31 13:58:56.315324 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.314049    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn9np\" (UniqueName: \"kubernetes.io/projected/bacfb744-b683-4b4d-a4e8-eefeaa418d71-kube-api-access-wn9np\") pod \"payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt\" (UID: \"bacfb744-b683-4b4d-a4e8-eefeaa418d71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"
Oct 31 13:58:56.328631 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.328591    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:56.470894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.424154    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacfb744-b683-4b4d-a4e8-eefeaa418d71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt\" (UID: \"bacfb744-b683-4b4d-a4e8-eefeaa418d71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"
Oct 31 13:58:56.470894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.424258    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn9np\" (UniqueName: \"kubernetes.io/projected/bacfb744-b683-4b4d-a4e8-eefeaa418d71-kube-api-access-wn9np\") pod \"payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt\" (UID: \"bacfb744-b683-4b4d-a4e8-eefeaa418d71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"
Oct 31 13:58:56.470894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.425667    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacfb744-b683-4b4d-a4e8-eefeaa418d71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt\" (UID: \"bacfb744-b683-4b4d-a4e8-eefeaa418d71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"
Oct 31 13:58:56.491455 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.491166    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"]
Oct 31 13:58:56.496574 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.496502    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdbe827a-0f8e-41f0-a10e-0893d404efbe" containerName="istio-proxy"
Oct 31 13:58:56.521372 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.518246    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"]
Oct 31 13:58:56.521372 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.518429    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"
Oct 31 13:58:56.604103 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.603588    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn9np\" (UniqueName: \"kubernetes.io/projected/bacfb744-b683-4b4d-a4e8-eefeaa418d71-kube-api-access-wn9np\") pod \"payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt\" (UID: \"bacfb744-b683-4b4d-a4e8-eefeaa418d71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"
Oct 31 13:58:56.628590 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.628553    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"
Oct 31 13:58:56.630137 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.630098    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e20d7a0-a20d-48fb-a666-c5c89ec7a927-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz\" (UID: \"0e20d7a0-a20d-48fb-a666-c5c89ec7a927\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"
Oct 31 13:58:56.630296 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.630178    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5pqc\" (UniqueName: \"kubernetes.io/projected/0e20d7a0-a20d-48fb-a666-c5c89ec7a927-kube-api-access-v5pqc\") pod \"payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz\" (UID: \"0e20d7a0-a20d-48fb-a666-c5c89ec7a927\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"
Oct 31 13:58:56.734124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.733276    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e20d7a0-a20d-48fb-a666-c5c89ec7a927-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz\" (UID: \"0e20d7a0-a20d-48fb-a666-c5c89ec7a927\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"
Oct 31 13:58:56.734124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.733367    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5pqc\" (UniqueName: \"kubernetes.io/projected/0e20d7a0-a20d-48fb-a666-c5c89ec7a927-kube-api-access-v5pqc\") pod \"payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz\" (UID: \"0e20d7a0-a20d-48fb-a666-c5c89ec7a927\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"
Oct 31 13:58:56.734124 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.734060    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e20d7a0-a20d-48fb-a666-c5c89ec7a927-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz\" (UID: \"0e20d7a0-a20d-48fb-a666-c5c89ec7a927\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"
Oct 31 13:58:56.742191 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.741927    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"
Oct 31 13:58:56.759895 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.759560    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5pqc\" (UniqueName: \"kubernetes.io/projected/0e20d7a0-a20d-48fb-a666-c5c89ec7a927-kube-api-access-v5pqc\") pod \"payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz\" (UID: \"0e20d7a0-a20d-48fb-a666-c5c89ec7a927\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"
Oct 31 13:58:56.769526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.767090    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"
Oct 31 13:58:56.779751 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.779497    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdbe827a-0f8e-41f0-a10e-0893d404efbe" path="/var/lib/kubelet/pods/fdbe827a-0f8e-41f0-a10e-0893d404efbe/volumes"
Oct 31 13:58:56.834463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.834425    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f-plugins-config-volume\") pod \"a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f\" (UID: \"a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f\") "
Oct 31 13:58:56.834944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.834906    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f-plugins-config-volume" pod "a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f" (UID: "a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:58:56.834944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.834854    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee72f133-ecfd-4910-8026-e0da592ead60-kube-api-access-8vcwv\" (UniqueName: \"kubernetes.io/projected/ee72f133-ecfd-4910-8026-e0da592ead60-kube-api-access-8vcwv\") pod \"ee72f133-ecfd-4910-8026-e0da592ead60\" (UID: \"ee72f133-ecfd-4910-8026-e0da592ead60\") "
Oct 31 13:58:56.835164 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.835148    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee72f133-ecfd-4910-8026-e0da592ead60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee72f133-ecfd-4910-8026-e0da592ead60-plugins-config-volume\") pod \"ee72f133-ecfd-4910-8026-e0da592ead60\" (UID: \"ee72f133-ecfd-4910-8026-e0da592ead60\") "
Oct 31 13:58:56.835237 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.835220    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f-kube-api-access-qtcqf\" (UniqueName: \"kubernetes.io/projected/a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f-kube-api-access-qtcqf\") pod \"a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f\" (UID: \"a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f\") "
Oct 31 13:58:56.836489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.836460    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee72f133-ecfd-4910-8026-e0da592ead60-plugins-config-volume" pod "ee72f133-ecfd-4910-8026-e0da592ead60" (UID: "ee72f133-ecfd-4910-8026-e0da592ead60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:58:56.837709 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.837026    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee72f133-ecfd-4910-8026-e0da592ead60-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:56.837709 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.837053    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:56.845383 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.845335    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee72f133-ecfd-4910-8026-e0da592ead60-kube-api-access-8vcwv" pod "ee72f133-ecfd-4910-8026-e0da592ead60" (UID: "ee72f133-ecfd-4910-8026-e0da592ead60"). InnerVolumeSpecName "kube-api-access-8vcwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:56.847528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.847490    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f-kube-api-access-qtcqf" pod "a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f" (UID: "a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f"). InnerVolumeSpecName "kube-api-access-qtcqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:58:56.927562 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.918578    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:56.938681 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.938631    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtcqf\" (UniqueName: \"kubernetes.io/projected/a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f-kube-api-access-qtcqf\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:56.938996 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.938958    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vcwv\" (UniqueName: \"kubernetes.io/projected/ee72f133-ecfd-4910-8026-e0da592ead60-kube-api-access-8vcwv\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:58:56.975070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:56.975025    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"
Oct 31 13:58:57.001724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.001692    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"]
Oct 31 13:58:57.002784 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:58:57.002745    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbacfb744_b683_4b4d_a4e8_eefeaa418d71.slice/crio-17056fce3292991dfb996e47e34ec9696ded1e75ebf13e915c3558e87259fb4c WatchSource:0}: Error finding container 17056fce3292991dfb996e47e34ec9696ded1e75ebf13e915c3558e87259fb4c: Status 404 returned error can't find the container with id 17056fce3292991dfb996e47e34ec9696ded1e75ebf13e915c3558e87259fb4c
Oct 31 13:58:57.126756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.126235    4428 generic.go:356] "Generic (PLEG): container finished" podID="a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f" containerID="9d360ce6fc453349b02be282d92ec3febea3b2c78a263cf224f3205a2d47b455" exitCode=0
Oct 31 13:58:57.126756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.126360    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr" event={"ID":"a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f","Type":"ContainerDied","Data":"9d360ce6fc453349b02be282d92ec3febea3b2c78a263cf224f3205a2d47b455"}
Oct 31 13:58:57.126756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.126417    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr" event={"ID":"a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f","Type":"ContainerDied","Data":"83f3d8a8bd8b84ac6e4cfe74a613caee2cbfcd0baf08f14fea037ab6f09193e7"}
Oct 31 13:58:57.126756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.126449    4428 scope.go:122] "RemoveContainer" containerID="9d360ce6fc453349b02be282d92ec3febea3b2c78a263cf224f3205a2d47b455"
Oct 31 13:58:57.126756 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.126604    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"
Oct 31 13:58:57.134947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.134165    4428 generic.go:356] "Generic (PLEG): container finished" podID="ee72f133-ecfd-4910-8026-e0da592ead60" containerID="ca30dec0fb80e96c9a4e758e406b69526a568c2e7cf94479b69f1566b60e81c6" exitCode=0
Oct 31 13:58:57.134947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.134300    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996" event={"ID":"ee72f133-ecfd-4910-8026-e0da592ead60","Type":"ContainerDied","Data":"ca30dec0fb80e96c9a4e758e406b69526a568c2e7cf94479b69f1566b60e81c6"}
Oct 31 13:58:57.134947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.134341    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996" event={"ID":"ee72f133-ecfd-4910-8026-e0da592ead60","Type":"ContainerDied","Data":"83b96b68bedffb70b2ed1cdad743f53369891d1e2115cc75d673a6fce194f6ef"}
Oct 31 13:58:57.134947 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.134437    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"
Oct 31 13:58:57.150540 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.149115    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt" event={"ID":"bacfb744-b683-4b4d-a4e8-eefeaa418d71","Type":"ContainerStarted","Data":"89e4713b3399c8f1fb2ab5be45a9db875aa234198c78679dbed3a2d9a1ee2200"}
Oct 31 13:58:57.150540 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.149182    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt" event={"ID":"bacfb744-b683-4b4d-a4e8-eefeaa418d71","Type":"ContainerStarted","Data":"17056fce3292991dfb996e47e34ec9696ded1e75ebf13e915c3558e87259fb4c"}
Oct 31 13:58:57.150540 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.150296    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"
Oct 31 13:58:57.157402 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.157382    4428 scope.go:122] "RemoveContainer" containerID="9d360ce6fc453349b02be282d92ec3febea3b2c78a263cf224f3205a2d47b455"
Oct 31 13:58:57.157750 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:58:57.157727    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d360ce6fc453349b02be282d92ec3febea3b2c78a263cf224f3205a2d47b455\": container with ID starting with 9d360ce6fc453349b02be282d92ec3febea3b2c78a263cf224f3205a2d47b455 not found: ID does not exist" containerID="9d360ce6fc453349b02be282d92ec3febea3b2c78a263cf224f3205a2d47b455"
Oct 31 13:58:57.157884 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.157759    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d360ce6fc453349b02be282d92ec3febea3b2c78a263cf224f3205a2d47b455"} err="failed to get container status \"9d360ce6fc453349b02be282d92ec3febea3b2c78a263cf224f3205a2d47b455\": rpc error: code = NotFound desc = could not find container \"9d360ce6fc453349b02be282d92ec3febea3b2c78a263cf224f3205a2d47b455\": container with ID starting with 9d360ce6fc453349b02be282d92ec3febea3b2c78a263cf224f3205a2d47b455 not found: ID does not exist"
Oct 31 13:58:57.157884 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.157784    4428 scope.go:122] "RemoveContainer" containerID="ca30dec0fb80e96c9a4e758e406b69526a568c2e7cf94479b69f1566b60e81c6"
Oct 31 13:58:57.176934 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.175518    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt" podStartSLOduration=1.175502832 podStartE2EDuration="1.175502832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:58:57.171974553 +0000 UTC m=+3334.570323177" watchObservedRunningTime="2026-10-31 13:58:57.175502832 +0000 UTC m=+3334.573851455"
Oct 31 13:58:57.182505 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.182478    4428 scope.go:122] "RemoveContainer" containerID="ca30dec0fb80e96c9a4e758e406b69526a568c2e7cf94479b69f1566b60e81c6"
Oct 31 13:58:57.182965 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:58:57.182935    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca30dec0fb80e96c9a4e758e406b69526a568c2e7cf94479b69f1566b60e81c6\": container with ID starting with ca30dec0fb80e96c9a4e758e406b69526a568c2e7cf94479b69f1566b60e81c6 not found: ID does not exist" containerID="ca30dec0fb80e96c9a4e758e406b69526a568c2e7cf94479b69f1566b60e81c6"
Oct 31 13:58:57.183047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.182979    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca30dec0fb80e96c9a4e758e406b69526a568c2e7cf94479b69f1566b60e81c6"} err="failed to get container status \"ca30dec0fb80e96c9a4e758e406b69526a568c2e7cf94479b69f1566b60e81c6\": rpc error: code = NotFound desc = could not find container \"ca30dec0fb80e96c9a4e758e406b69526a568c2e7cf94479b69f1566b60e81c6\": container with ID starting with ca30dec0fb80e96c9a4e758e406b69526a568c2e7cf94479b69f1566b60e81c6 not found: ID does not exist"
Oct 31 13:58:57.191844 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.191811    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"]
Oct 31 13:58:57.200354 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.200320    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-612b46-85d6fdfb9-vlqkr"]
Oct 31 13:58:57.211242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.211211    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"]
Oct 31 13:58:57.220451 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.220413    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-612b46-77485d8f9f-25996"]
Oct 31 13:58:57.323964 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.307793    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"]
Oct 31 13:58:57.335711 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:58:57.327498    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e20d7a0_a20d_48fb_a666_c5c89ec7a927.slice/crio-cf5e8042692d90e6cee6caab9ebafbcee19a9d4eb6705a5a3dc5ee1a00e2b852 WatchSource:0}: Error finding container cf5e8042692d90e6cee6caab9ebafbcee19a9d4eb6705a5a3dc5ee1a00e2b852: Status 404 returned error can't find the container with id cf5e8042692d90e6cee6caab9ebafbcee19a9d4eb6705a5a3dc5ee1a00e2b852
Oct 31 13:58:57.873352 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:57.873302    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:58.159907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:58.159803    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz" event={"ID":"0e20d7a0-a20d-48fb-a666-c5c89ec7a927","Type":"ContainerStarted","Data":"1b980d1002f58f4d44cb0ad875b7db658607cf113a782d703712ced96ca8e352"}
Oct 31 13:58:58.159907 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:58.159862    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz" event={"ID":"0e20d7a0-a20d-48fb-a666-c5c89ec7a927","Type":"ContainerStarted","Data":"cf5e8042692d90e6cee6caab9ebafbcee19a9d4eb6705a5a3dc5ee1a00e2b852"}
Oct 31 13:58:58.160430 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:58.159938    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"
Oct 31 13:58:58.185337 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:58.185282    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz" podStartSLOduration=2.185264822 podStartE2EDuration="2.185264822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:58:58.180761009 +0000 UTC m=+3335.579109636" watchObservedRunningTime="2026-10-31 13:58:58.185264822 +0000 UTC m=+3335.583613446"
Oct 31 13:58:58.758817 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:58.758783    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f" path="/var/lib/kubelet/pods/a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f/volumes"
Oct 31 13:58:58.759398 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:58.759374    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee72f133-ecfd-4910-8026-e0da592ead60" path="/var/lib/kubelet/pods/ee72f133-ecfd-4910-8026-e0da592ead60/volumes"
Oct 31 13:58:58.886340 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:58.886296    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:58:59.728847 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:58:59.726859    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:00.062655 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:00.062623    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"
Oct 31 13:59:00.074383 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:00.074355    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"
Oct 31 13:59:00.177749 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:00.177706    4428 generic.go:356] "Generic (PLEG): container finished" podID="6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" containerID="460cfc5b72ecfb55a6b52dbb7b554542b0791c37027e4312fe093a7db81604ef" exitCode=0
Oct 31 13:59:00.178011 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:00.177779    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj" event={"ID":"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d","Type":"ContainerDied","Data":"460cfc5b72ecfb55a6b52dbb7b554542b0791c37027e4312fe093a7db81604ef"}
Oct 31 13:59:00.178372 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:00.178354    4428 scope.go:122] "RemoveContainer" containerID="460cfc5b72ecfb55a6b52dbb7b554542b0791c37027e4312fe093a7db81604ef"
Oct 31 13:59:00.180311 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:00.180283    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:59:00.180382 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:00.180355    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:59:00.180421 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:00.180388    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:59:00.695916 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:00.694057    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:00.874776 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:00.874718    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:01.035069 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:01.035033    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:01.222905 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:01.222461    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj" event={"ID":"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d","Type":"ContainerStarted","Data":"0a96c1ca088318c212b57afa1adb59adf5f2fd39b61612c5bbb7eab5d1d04d1d"}
Oct 31 13:59:01.599086 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:01.597251    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:59:01.603097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:01.602987    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body=
Oct 31 13:59:01.603097 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:01.603046    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj" podUID="6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused"
Oct 31 13:59:01.620894 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:01.618267    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:02.383489 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:02.383442    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:02.589500 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:02.589458    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body=
Oct 31 13:59:02.589727 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:02.589526    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj" podUID="6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused"
Oct 31 13:59:03.338893 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:03.337837    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:03.590327 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:03.590230    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body=
Oct 31 13:59:03.590327 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:03.590294    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj" podUID="6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused"
Oct 31 13:59:03.709613 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:03.709574    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:04.081358 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.081317    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"]
Oct 31 13:59:04.082642 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.082610    4428 cpu_manager.go:414] "containerMap: removing container" podUID="a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f" containerName="payload-processing"
Oct 31 13:59:04.082642 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.082642    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f" containerName="payload-processing"
Oct 31 13:59:04.082911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.082746    4428 cpu_manager.go:414] "containerMap: removing container" podUID="ee72f133-ecfd-4910-8026-e0da592ead60" containerName="payload-pre-processing"
Oct 31 13:59:04.082911 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.082756    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee72f133-ecfd-4910-8026-e0da592ead60" containerName="payload-pre-processing"
Oct 31 13:59:04.083036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.082960    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee72f133-ecfd-4910-8026-e0da592ead60" containerName="payload-pre-processing"
Oct 31 13:59:04.083036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.082983    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22cfa98-6a24-4aea-89d1-9c3ef6a26f9f" containerName="payload-processing"
Oct 31 13:59:04.096004 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.095938    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"]
Oct 31 13:59:04.096251 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.096094    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.123843 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.123802    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f9ca72-openshift-default-dockercfg-wk4k4\""
Oct 31 13:59:04.172183 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.171996    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:04.190160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.188744    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-credential-socket\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.190160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.188823    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-token\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.190160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.188891    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-envoy\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.190160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.188967    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2da70ae-abad-42d5-be21-0fd2b83848e0-istiod-ca-cert\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.190160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.189014    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-podinfo\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.190160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.189147    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-data\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.190160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.189182    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-workload-socket\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.190160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.189220    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrlpd\" (UniqueName: \"kubernetes.io/projected/c2da70ae-abad-42d5-be21-0fd2b83848e0-kube-api-access-qrlpd\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.190160 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.189296    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-workload-certs\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.296995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.293848    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-credential-socket\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.296995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.293962    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-token\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.296995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.294007    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-envoy\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.296995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.294078    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2da70ae-abad-42d5-be21-0fd2b83848e0-istiod-ca-cert\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.296995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.294126    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-podinfo\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.296995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.294252    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-data\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.296995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.294288    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-workload-socket\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.296995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.294324    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrlpd\" (UniqueName: \"kubernetes.io/projected/c2da70ae-abad-42d5-be21-0fd2b83848e0-kube-api-access-qrlpd\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.296995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.294397    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-workload-certs\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.296995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.294961    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-workload-certs\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.296995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.295220    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-credential-socket\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.343994 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.335506    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-data\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.353034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.342796    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-workload-socket\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.353329 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.343780    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2da70ae-abad-42d5-be21-0fd2b83848e0-istiod-ca-cert\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.353501 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.348523    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-envoy\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.375822 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.375787    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-podinfo\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.377005 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.376398    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:04.377982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.377957    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrlpd\" (UniqueName: \"kubernetes.io/projected/c2da70ae-abad-42d5-be21-0fd2b83848e0-kube-api-access-qrlpd\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.381049 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.381025    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-token\") pod \"e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") " pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.424435 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.424399    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:04.609114 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.594415    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body=
Oct 31 13:59:04.609114 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.594476    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj" podUID="6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused"
Oct 31 13:59:04.637241 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.632035    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:04.870146 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:59:04.870043    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2da70ae_abad_42d5_be21_0fd2b83848e0.slice/crio-ce5f75d96d644a40dabddc77a5d53632b0cef229142249cf6e226403e98f1154 WatchSource:0}: Error finding container ce5f75d96d644a40dabddc77a5d53632b0cef229142249cf6e226403e98f1154: Status 404 returned error can't find the container with id ce5f75d96d644a40dabddc77a5d53632b0cef229142249cf6e226403e98f1154
Oct 31 13:59:04.871846 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.871819    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"]
Oct 31 13:59:04.873604 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.873572    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:59:04.873687 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.873641    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:59:04.873687 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:04.873671    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 13:59:05.141473 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:05.141380    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:05.374150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:05.374111    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt" event={"ID":"c2da70ae-abad-42d5-be21-0fd2b83848e0","Type":"ContainerStarted","Data":"36e54d51c0030b7bffd4c8fe58c4d90e35019d908827fbbca0b30fa559097371"}
Oct 31 13:59:05.374370 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:05.374172    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt" event={"ID":"c2da70ae-abad-42d5-be21-0fd2b83848e0","Type":"ContainerStarted","Data":"ce5f75d96d644a40dabddc77a5d53632b0cef229142249cf6e226403e98f1154"}
Oct 31 13:59:05.400528 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:05.400404    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt" podStartSLOduration=1.400380716 podStartE2EDuration="1.400380716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:59:05.398363931 +0000 UTC m=+3342.796712557" watchObservedRunningTime="2026-10-31 13:59:05.400380716 +0000 UTC m=+3342.798729341"
Oct 31 13:59:05.425413 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:05.425376    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:05.427240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:05.427214    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 31 13:59:05.427394 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:05.427269    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt" podUID="c2da70ae-abad-42d5-be21-0fd2b83848e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 31 13:59:05.592433 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:05.592397    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body=
Oct 31 13:59:05.592763 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:05.592717    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj" podUID="6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused"
Oct 31 13:59:05.909100 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:05.900538    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:06.074340 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:06.074304    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:59:06.431222 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:06.431193    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:06.724721 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:06.724684    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:59:06.727070 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:06.727044    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 13:59:06.839136 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:06.836808    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:07.390957 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:07.390907    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:07.391808 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:07.391789    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 13:59:07.798377 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:07.798336    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:08.059952 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:08.059214    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:09.171849 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.171818    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"
Oct 31 13:59:09.172388 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.172092    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"
Oct 31 13:59:09.212998 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.212967    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:09.275860 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.275821    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"]
Oct 31 13:59:09.280793 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.280767    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"
Oct 31 13:59:09.292683 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.292651    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"]
Oct 31 13:59:09.313749 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.313713    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/509c1833-4923-4718-9661-1cf9a861fdbb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6\" (UID: \"509c1833-4923-4718-9661-1cf9a861fdbb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"
Oct 31 13:59:09.314118 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.314089    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hpfq\" (UniqueName: \"kubernetes.io/projected/509c1833-4923-4718-9661-1cf9a861fdbb-kube-api-access-2hpfq\") pod \"payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6\" (UID: \"509c1833-4923-4718-9661-1cf9a861fdbb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"
Oct 31 13:59:09.427956 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.425596    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:09.442898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.428615    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f9ca72-dockercfg-hvzgx\""
Oct 31 13:59:09.442898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.428714    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f9ca72\""
Oct 31 13:59:09.442898 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.439047    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hpfq\" (UniqueName: \"kubernetes.io/projected/509c1833-4923-4718-9661-1cf9a861fdbb-kube-api-access-2hpfq\") pod \"payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6\" (UID: \"509c1833-4923-4718-9661-1cf9a861fdbb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"
Oct 31 13:59:09.497215 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.462423    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/509c1833-4923-4718-9661-1cf9a861fdbb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6\" (UID: \"509c1833-4923-4718-9661-1cf9a861fdbb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"
Oct 31 13:59:09.497215 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.463674    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/509c1833-4923-4718-9661-1cf9a861fdbb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6\" (UID: \"509c1833-4923-4718-9661-1cf9a861fdbb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"
Oct 31 13:59:09.507526 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.507468    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hpfq\" (UniqueName: \"kubernetes.io/projected/509c1833-4923-4718-9661-1cf9a861fdbb-kube-api-access-2hpfq\") pod \"payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6\" (UID: \"509c1833-4923-4718-9661-1cf9a861fdbb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"
Oct 31 13:59:09.593552 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.593509    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:09.622359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.622304    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"
Oct 31 13:59:09.941036 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:09.941002    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"]
Oct 31 13:59:09.941439 ip-10-0-136-168 kubenswrapper[4428]: W1031 13:59:09.941414    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509c1833_4923_4718_9661_1cf9a861fdbb.slice/crio-9bdc73150dc8b0412c3f6c2097a151d2acbba5519e577abb2583bce0421c8899 WatchSource:0}: Error finding container 9bdc73150dc8b0412c3f6c2097a151d2acbba5519e577abb2583bce0421c8899: Status 404 returned error can't find the container with id 9bdc73150dc8b0412c3f6c2097a151d2acbba5519e577abb2583bce0421c8899
Oct 31 13:59:10.391724 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.389779    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main" containerID="cri-o://3f3782d8b48f18b56dd672c86e1a1b6fa8b9193efb564bd1d12664266ae18012" gracePeriod=45
Oct 31 13:59:10.516944 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.504735    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:10.535401 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.535364    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6" event={"ID":"509c1833-4923-4718-9661-1cf9a861fdbb","Type":"ContainerStarted","Data":"3faeae15d645c7c7895f8cda8d584a7c8355af3488d652d88c4a11f00ba8b45e"}
Oct 31 13:59:10.535529 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.535435    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6" event={"ID":"509c1833-4923-4718-9661-1cf9a861fdbb","Type":"ContainerStarted","Data":"9bdc73150dc8b0412c3f6c2097a151d2acbba5519e577abb2583bce0421c8899"}
Oct 31 13:59:10.535576 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.535559    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"
Oct 31 13:59:10.562896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.561165    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6" podStartSLOduration=1.561150644 podStartE2EDuration="1.561150644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:59:10.559030616 +0000 UTC m=+3347.957379243" watchObservedRunningTime="2026-10-31 13:59:10.561150644 +0000 UTC m=+3347.959499267"
Oct 31 13:59:10.721801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.721769    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:59:10.795287 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.795254    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-dshm\") pod \"4f2ff4c9-6250-41b6-b780-e5f189139e59\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") "
Oct 31 13:59:10.795508 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.795325    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-model-cache\") pod \"4f2ff4c9-6250-41b6-b780-e5f189139e59\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") "
Oct 31 13:59:10.795508 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.795388    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f2ff4c9-6250-41b6-b780-e5f189139e59-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2ff4c9-6250-41b6-b780-e5f189139e59-tls-certs\") pod \"4f2ff4c9-6250-41b6-b780-e5f189139e59\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") "
Oct 31 13:59:10.795508 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.795457    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-tmp-dir\") pod \"4f2ff4c9-6250-41b6-b780-e5f189139e59\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") "
Oct 31 13:59:10.795714 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.795530    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-home\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-home\") pod \"4f2ff4c9-6250-41b6-b780-e5f189139e59\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") "
Oct 31 13:59:10.795798 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.795776    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2ff4c9-6250-41b6-b780-e5f189139e59-kube-api-access-5gmkl\" (UniqueName: \"kubernetes.io/projected/4f2ff4c9-6250-41b6-b780-e5f189139e59-kube-api-access-5gmkl\") pod \"4f2ff4c9-6250-41b6-b780-e5f189139e59\" (UID: \"4f2ff4c9-6250-41b6-b780-e5f189139e59\") "
Oct 31 13:59:10.795955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.795930    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-tmp-dir" pod "4f2ff4c9-6250-41b6-b780-e5f189139e59" (UID: "4f2ff4c9-6250-41b6-b780-e5f189139e59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:59:10.796159 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.796138    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-home" pod "4f2ff4c9-6250-41b6-b780-e5f189139e59" (UID: "4f2ff4c9-6250-41b6-b780-e5f189139e59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:59:10.796396 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.796369    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-model-cache" pod "4f2ff4c9-6250-41b6-b780-e5f189139e59" (UID: "4f2ff4c9-6250-41b6-b780-e5f189139e59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:59:10.796686 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.796656    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:10.796913 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.796775    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:10.797001 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.796916    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:10.807706 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.807681    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-dshm" pod "4f2ff4c9-6250-41b6-b780-e5f189139e59" (UID: "4f2ff4c9-6250-41b6-b780-e5f189139e59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:59:10.809065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.808118    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2ff4c9-6250-41b6-b780-e5f189139e59-kube-api-access-5gmkl" pod "4f2ff4c9-6250-41b6-b780-e5f189139e59" (UID: "4f2ff4c9-6250-41b6-b780-e5f189139e59"). InnerVolumeSpecName "kube-api-access-5gmkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:59:10.809065 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.809027    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2ff4c9-6250-41b6-b780-e5f189139e59-tls-certs" pod "4f2ff4c9-6250-41b6-b780-e5f189139e59" (UID: "4f2ff4c9-6250-41b6-b780-e5f189139e59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:59:10.899342 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.899305    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gmkl\" (UniqueName: \"kubernetes.io/projected/4f2ff4c9-6250-41b6-b780-e5f189139e59-kube-api-access-5gmkl\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:10.899342 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.899342    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2ff4c9-6250-41b6-b780-e5f189139e59-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:10.899342 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:10.899359    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2ff4c9-6250-41b6-b780-e5f189139e59-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:11.570559 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:11.570517    4428 generic.go:356] "Generic (PLEG): container finished" podID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerID="3f3782d8b48f18b56dd672c86e1a1b6fa8b9193efb564bd1d12664266ae18012" exitCode=0
Oct 31 13:59:11.571076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:11.570605    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"
Oct 31 13:59:11.571076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:11.570598    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" event={"ID":"4f2ff4c9-6250-41b6-b780-e5f189139e59","Type":"ContainerDied","Data":"3f3782d8b48f18b56dd672c86e1a1b6fa8b9193efb564bd1d12664266ae18012"}
Oct 31 13:59:11.571076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:11.570743    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642" event={"ID":"4f2ff4c9-6250-41b6-b780-e5f189139e59","Type":"ContainerDied","Data":"1707362bd6b181b2ea9f951f10a6b420291035276b9a7443569d901ce1d40f90"}
Oct 31 13:59:11.571076 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:11.570774    4428 scope.go:122] "RemoveContainer" containerID="3f3782d8b48f18b56dd672c86e1a1b6fa8b9193efb564bd1d12664266ae18012"
Oct 31 13:59:11.583242 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:11.583222    4428 scope.go:122] "RemoveContainer" containerID="3f3782d8b48f18b56dd672c86e1a1b6fa8b9193efb564bd1d12664266ae18012"
Oct 31 13:59:11.583564 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:59:11.583545    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3782d8b48f18b56dd672c86e1a1b6fa8b9193efb564bd1d12664266ae18012\": container with ID starting with 3f3782d8b48f18b56dd672c86e1a1b6fa8b9193efb564bd1d12664266ae18012 not found: ID does not exist" containerID="3f3782d8b48f18b56dd672c86e1a1b6fa8b9193efb564bd1d12664266ae18012"
Oct 31 13:59:11.583633 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:11.583572    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3782d8b48f18b56dd672c86e1a1b6fa8b9193efb564bd1d12664266ae18012"} err="failed to get container status \"3f3782d8b48f18b56dd672c86e1a1b6fa8b9193efb564bd1d12664266ae18012\": rpc error: code = NotFound desc = could not find container \"3f3782d8b48f18b56dd672c86e1a1b6fa8b9193efb564bd1d12664266ae18012\": container with ID starting with 3f3782d8b48f18b56dd672c86e1a1b6fa8b9193efb564bd1d12664266ae18012 not found: ID does not exist"
Oct 31 13:59:11.586212 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:11.586187    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:11.600450 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:11.600415    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"]
Oct 31 13:59:11.604345 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:11.604322    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/simulator-w0-7b3ddd-kserve-56d787bdc7-hx642"]
Oct 31 13:59:12.514458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:12.514287    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:12.757150 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:12.757114    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" path="/var/lib/kubelet/pods/4f2ff4c9-6250-41b6-b780-e5f189139e59/volumes"
Oct 31 13:59:13.312892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:13.305132    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:14.068240 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:14.066695    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:14.834827 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:14.834793    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:15.253618 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:15.253585    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:16.199359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:16.199321    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:16.365431 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:16.365399    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:17.540546 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:17.540505    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:18.334646 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:18.326674    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:18.742862 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:59:18.742818    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:59:19.247838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:19.247800    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:20.286273 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:20.286211    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:21.099647 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:21.099613    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:21.574750 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:21.574720    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"
Oct 31 13:59:21.985471 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:21.985364    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:22.227447 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:22.227420    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w9wjk_e328741d-31fd-40db-8cbb-8c9b518f4cde/payload-processing/0.log"
Oct 31 13:59:22.413808 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:22.413653    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w9wjk_e328741d-31fd-40db-8cbb-8c9b518f4cde/payload-processing/0.log"
Oct 31 13:59:22.583073 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:22.583046    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b_66b6b2a9-e6b8-4848-9fb7-4886a7db34e4/payload-processing/0.log"
Oct 31 13:59:22.723094 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:22.723062    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:22.844951 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:22.843816    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b_66b6b2a9-e6b8-4848-9fb7-4886a7db34e4/payload-processing/0.log"
Oct 31 13:59:23.745896 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:23.745807    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:24.529116 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:24.528943    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:25.385378 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:25.385305    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:26.141878 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:26.141810    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:26.901110 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:26.901079    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:27.055605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:27.055574    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:27.211331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:27.211299    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:27.819677 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:27.819645    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:28.803861 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:28.803825    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:29.835597 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:29.835542    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:30.609316 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:30.609269    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:31.338198 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:31.338143    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:32.385161 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:32.385116    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:32.567439 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:32.567402    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:32.837892 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:32.837448    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:33.087058 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:33.087018    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:35.242749 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:35.242622    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:36.053128 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:36.053089    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:36.997153 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:36.997116    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:37.557173 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:37.557128    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"]
Oct 31 13:59:37.557479 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:37.557445    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj" podUID="82f12bdb-2c78-4081-9cfc-653a252ab25b" containerName="maas-api" containerID="cri-o://981bde87e95607822b7e264bf4a92c36264f924b7715457574173e03c098227a" gracePeriod=30
Oct 31 13:59:37.640163 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:37.640096    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:37.809553 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:37.809471    4428 generic.go:356] "Generic (PLEG): container finished" podID="82f12bdb-2c78-4081-9cfc-653a252ab25b" containerID="981bde87e95607822b7e264bf4a92c36264f924b7715457574173e03c098227a" exitCode=0
Oct 31 13:59:37.809553 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:37.809535    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj" event={"ID":"82f12bdb-2c78-4081-9cfc-653a252ab25b","Type":"ContainerDied","Data":"981bde87e95607822b7e264bf4a92c36264f924b7715457574173e03c098227a"}
Oct 31 13:59:37.882965 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:37.860811    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:38.029995 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.029961    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:38.076047 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.075949    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:59:38.137263 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.137224    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82f12bdb-2c78-4081-9cfc-653a252ab25b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/82f12bdb-2c78-4081-9cfc-653a252ab25b-metrics-certs\") pod \"82f12bdb-2c78-4081-9cfc-653a252ab25b\" (UID: \"82f12bdb-2c78-4081-9cfc-653a252ab25b\") "
Oct 31 13:59:38.137477 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.137352    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f12bdb-2c78-4081-9cfc-653a252ab25b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f12bdb-2c78-4081-9cfc-653a252ab25b-openshift-service-ca\") pod \"82f12bdb-2c78-4081-9cfc-653a252ab25b\" (UID: \"82f12bdb-2c78-4081-9cfc-653a252ab25b\") "
Oct 31 13:59:38.137477 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.137424    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f12bdb-2c78-4081-9cfc-653a252ab25b-kube-api-access-bzmkv\" (UniqueName: \"kubernetes.io/projected/82f12bdb-2c78-4081-9cfc-653a252ab25b-kube-api-access-bzmkv\") pod \"82f12bdb-2c78-4081-9cfc-653a252ab25b\" (UID: \"82f12bdb-2c78-4081-9cfc-653a252ab25b\") "
Oct 31 13:59:38.137623 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.137559    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82f12bdb-2c78-4081-9cfc-653a252ab25b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82f12bdb-2c78-4081-9cfc-653a252ab25b-maas-api-tls\") pod \"82f12bdb-2c78-4081-9cfc-653a252ab25b\" (UID: \"82f12bdb-2c78-4081-9cfc-653a252ab25b\") "
Oct 31 13:59:38.137777 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.137745    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f12bdb-2c78-4081-9cfc-653a252ab25b-openshift-service-ca" pod "82f12bdb-2c78-4081-9cfc-653a252ab25b" (UID: "82f12bdb-2c78-4081-9cfc-653a252ab25b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:59:38.138534 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.138500    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f12bdb-2c78-4081-9cfc-653a252ab25b-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:38.140803 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.140766    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f12bdb-2c78-4081-9cfc-653a252ab25b-maas-api-tls" pod "82f12bdb-2c78-4081-9cfc-653a252ab25b" (UID: "82f12bdb-2c78-4081-9cfc-653a252ab25b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:59:38.140972 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.140827    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f12bdb-2c78-4081-9cfc-653a252ab25b-kube-api-access-bzmkv" pod "82f12bdb-2c78-4081-9cfc-653a252ab25b" (UID: "82f12bdb-2c78-4081-9cfc-653a252ab25b"). InnerVolumeSpecName "kube-api-access-bzmkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:59:38.141324 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.141299    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f12bdb-2c78-4081-9cfc-653a252ab25b-metrics-certs" pod "82f12bdb-2c78-4081-9cfc-653a252ab25b" (UID: "82f12bdb-2c78-4081-9cfc-653a252ab25b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:59:38.227909 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.227831    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:38.240211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.240174    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82f12bdb-2c78-4081-9cfc-653a252ab25b-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:38.240211 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.240211    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82f12bdb-2c78-4081-9cfc-653a252ab25b-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:38.240412 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.240225    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzmkv\" (UniqueName: \"kubernetes.io/projected/82f12bdb-2c78-4081-9cfc-653a252ab25b-kube-api-access-bzmkv\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:38.388462 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.388377    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:38.740331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.740296    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:38.817838 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.817811    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"
Oct 31 13:59:38.818045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.817844    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj" event={"ID":"82f12bdb-2c78-4081-9cfc-653a252ab25b","Type":"ContainerDied","Data":"f8e6ab7ed6bf9e2ea9f3f15c0653d43ad87a59fcb3de80fd88b47f0bc337225e"}
Oct 31 13:59:38.818045 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.817950    4428 scope.go:122] "RemoveContainer" containerID="981bde87e95607822b7e264bf4a92c36264f924b7715457574173e03c098227a"
Oct 31 13:59:38.843359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.843330    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"]
Oct 31 13:59:38.851815 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:38.851785    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj"]
Oct 31 13:59:39.270992 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:39.270953    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:40.044643 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:40.044608    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:40.264384 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:40.264347    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:40.571156 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:40.571120    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:40.758018 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:40.757986    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f12bdb-2c78-4081-9cfc-653a252ab25b" path="/var/lib/kubelet/pods/82f12bdb-2c78-4081-9cfc-653a252ab25b/volumes"
Oct 31 13:59:40.787651 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:40.787618    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:40.824463 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:40.812992    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e50b22-866dfb9f6d-2vdwj" podUID="82f12bdb-2c78-4081-9cfc-653a252ab25b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.1.10:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:59:41.195809 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:41.195716    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:41.703805 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:41.703766    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:41.866605 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:41.866566    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:42.044410 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:42.044378    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:42.910897 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:42.908924    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:43.835252 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:43.835214    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:44.644403 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:44.644370    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:45.598471 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:45.598436    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:46.185760 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:46.185722    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:46.362234 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:46.362202    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:46.523955 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:46.522458    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:47.393318 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:47.393268    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:48.437260 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:48.437209    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:48.598900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:48.598824    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:48.743085 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:59:48.743043    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 13:59:48.761002 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:48.760970    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:49.897331 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:49.897289    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:50.776148 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:50.776092    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:51.632799 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:51.632751    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:52.526185 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:52.522303    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:53.328139 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:53.325185    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:53.389684 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:53.389647    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"]
Oct 31 13:59:53.519367 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:53.519329    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:53.695948 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:53.693370    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:53.864458 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:53.864423    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:54.515184 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:54.515148    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:55.392465 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:55.392423    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:56.504903 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:56.504809    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:57.434419 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:57.434363    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:58.228627 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:58.228592    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:58.390723 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:58.390681    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6" podUID="509c1833-4923-4718-9661-1cf9a861fdbb" containerName="payload-processing" containerID="cri-o://3faeae15d645c7c7895f8cda8d584a7c8355af3488d652d88c4a11f00ba8b45e" gracePeriod=25
Oct 31 13:59:58.700707 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:58.700678    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"
Oct 31 13:59:58.828801 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:58.828716    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/509c1833-4923-4718-9661-1cf9a861fdbb-kube-api-access-2hpfq\" (UniqueName: \"kubernetes.io/projected/509c1833-4923-4718-9661-1cf9a861fdbb-kube-api-access-2hpfq\") pod \"509c1833-4923-4718-9661-1cf9a861fdbb\" (UID: \"509c1833-4923-4718-9661-1cf9a861fdbb\") "
Oct 31 13:59:58.829037 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:58.828821    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/509c1833-4923-4718-9661-1cf9a861fdbb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/509c1833-4923-4718-9661-1cf9a861fdbb-plugins-config-volume\") pod \"509c1833-4923-4718-9661-1cf9a861fdbb\" (UID: \"509c1833-4923-4718-9661-1cf9a861fdbb\") "
Oct 31 13:59:58.829248 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:58.829226    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/509c1833-4923-4718-9661-1cf9a861fdbb-plugins-config-volume" pod "509c1833-4923-4718-9661-1cf9a861fdbb" (UID: "509c1833-4923-4718-9661-1cf9a861fdbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:59:58.829390 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:58.829374    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/509c1833-4923-4718-9661-1cf9a861fdbb-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:58.831334 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:58.831307    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/509c1833-4923-4718-9661-1cf9a861fdbb-kube-api-access-2hpfq" pod "509c1833-4923-4718-9661-1cf9a861fdbb" (UID: "509c1833-4923-4718-9661-1cf9a861fdbb"). InnerVolumeSpecName "kube-api-access-2hpfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:59:58.931359 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:58.931323    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hpfq\" (UniqueName: \"kubernetes.io/projected/509c1833-4923-4718-9661-1cf9a861fdbb-kube-api-access-2hpfq\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:59:59.119982 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:59.113517    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 13:59:59.140900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:59.137652    4428 generic.go:356] "Generic (PLEG): container finished" podID="509c1833-4923-4718-9661-1cf9a861fdbb" containerID="3faeae15d645c7c7895f8cda8d584a7c8355af3488d652d88c4a11f00ba8b45e" exitCode=0
Oct 31 13:59:59.140900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:59.137907    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6" event={"ID":"509c1833-4923-4718-9661-1cf9a861fdbb","Type":"ContainerDied","Data":"3faeae15d645c7c7895f8cda8d584a7c8355af3488d652d88c4a11f00ba8b45e"}
Oct 31 13:59:59.140900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:59.137959    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6" event={"ID":"509c1833-4923-4718-9661-1cf9a861fdbb","Type":"ContainerDied","Data":"9bdc73150dc8b0412c3f6c2097a151d2acbba5519e577abb2583bce0421c8899"}
Oct 31 13:59:59.140900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:59.137985    4428 scope.go:122] "RemoveContainer" containerID="3faeae15d645c7c7895f8cda8d584a7c8355af3488d652d88c4a11f00ba8b45e"
Oct 31 13:59:59.140900 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:59.138152    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"
Oct 31 13:59:59.162172 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:59.162144    4428 scope.go:122] "RemoveContainer" containerID="3faeae15d645c7c7895f8cda8d584a7c8355af3488d652d88c4a11f00ba8b45e"
Oct 31 13:59:59.162531 ip-10-0-136-168 kubenswrapper[4428]: E1031 13:59:59.162497    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3faeae15d645c7c7895f8cda8d584a7c8355af3488d652d88c4a11f00ba8b45e\": container with ID starting with 3faeae15d645c7c7895f8cda8d584a7c8355af3488d652d88c4a11f00ba8b45e not found: ID does not exist" containerID="3faeae15d645c7c7895f8cda8d584a7c8355af3488d652d88c4a11f00ba8b45e"
Oct 31 13:59:59.162635 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:59.162555    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3faeae15d645c7c7895f8cda8d584a7c8355af3488d652d88c4a11f00ba8b45e"} err="failed to get container status \"3faeae15d645c7c7895f8cda8d584a7c8355af3488d652d88c4a11f00ba8b45e\": rpc error: code = NotFound desc = could not find container \"3faeae15d645c7c7895f8cda8d584a7c8355af3488d652d88c4a11f00ba8b45e\": container with ID starting with 3faeae15d645c7c7895f8cda8d584a7c8355af3488d652d88c4a11f00ba8b45e not found: ID does not exist"
Oct 31 13:59:59.177530 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:59.177495    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"]
Oct 31 13:59:59.193034 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:59.193001    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-f5cbf94c5-hd9n6"]
Oct 31 13:59:59.898623 ip-10-0-136-168 kubenswrapper[4428]: I1031 13:59:59.898589    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:00.149491 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.149396    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82"]
Oct 31 14:00:00.150224 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.150202    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main"
Oct 31 14:00:00.150224 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.150221    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main"
Oct 31 14:00:00.150428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.150246    4428 cpu_manager.go:414] "containerMap: removing container" podUID="509c1833-4923-4718-9661-1cf9a861fdbb" containerName="payload-processing"
Oct 31 14:00:00.150428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.150251    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="509c1833-4923-4718-9661-1cf9a861fdbb" containerName="payload-processing"
Oct 31 14:00:00.150428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.150263    4428 cpu_manager.go:414] "containerMap: removing container" podUID="82f12bdb-2c78-4081-9cfc-653a252ab25b" containerName="maas-api"
Oct 31 14:00:00.150428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.150270    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f12bdb-2c78-4081-9cfc-653a252ab25b" containerName="maas-api"
Oct 31 14:00:00.150428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.150381    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="509c1833-4923-4718-9661-1cf9a861fdbb" containerName="payload-processing"
Oct 31 14:00:00.150428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.150390    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f12bdb-2c78-4081-9cfc-653a252ab25b" containerName="maas-api"
Oct 31 14:00:00.150428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.150402    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2ff4c9-6250-41b6-b780-e5f189139e59" containerName="main"
Oct 31 14:00:00.155557 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.155535    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82"
Oct 31 14:00:00.173367 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.171975    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82"]
Oct 31 14:00:00.182800 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.182762    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n"]
Oct 31 14:00:00.196029 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.195897    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg"]
Oct 31 14:00:00.197488 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.197391    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n"
Oct 31 14:00:00.219008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.218380    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp"]
Oct 31 14:00:00.219954 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.219930    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg"
Oct 31 14:00:00.270934 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.254373    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49565e4-99d7-4a99-a912-047c19c4b8b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82\" (UID: \"b49565e4-99d7-4a99-a912-047c19c4b8b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82"
Oct 31 14:00:00.270934 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.254469    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/124e886b-210d-465f-b6a2-5dcd03b03457-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg\" (UID: \"124e886b-210d-465f-b6a2-5dcd03b03457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg"
Oct 31 14:00:00.270934 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.254527    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hcfc\" (UniqueName: \"kubernetes.io/projected/124e886b-210d-465f-b6a2-5dcd03b03457-kube-api-access-2hcfc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg\" (UID: \"124e886b-210d-465f-b6a2-5dcd03b03457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg"
Oct 31 14:00:00.270934 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.254577    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7rjf\" (UniqueName: \"kubernetes.io/projected/0fee8a48-bc4e-4df9-b4db-354d2f577fd0-kube-api-access-f7rjf\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n\" (UID: \"0fee8a48-bc4e-4df9-b4db-354d2f577fd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n"
Oct 31 14:00:00.270934 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.254730    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fee8a48-bc4e-4df9-b4db-354d2f577fd0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n\" (UID: \"0fee8a48-bc4e-4df9-b4db-354d2f577fd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n"
Oct 31 14:00:00.270934 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.254791    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxlrx\" (UniqueName: \"kubernetes.io/projected/b49565e4-99d7-4a99-a912-047c19c4b8b6-kube-api-access-zxlrx\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82\" (UID: \"b49565e4-99d7-4a99-a912-047c19c4b8b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82"
Oct 31 14:00:00.270934 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.262247    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:00:00.270934 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.264403    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n"]
Oct 31 14:00:00.270934 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.264552    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp"
Oct 31 14:00:00.277806 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.277168    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg"]
Oct 31 14:00:00.283000 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.282966    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp"]
Oct 31 14:00:00.288096 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.288056    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q"]
Oct 31 14:00:00.294929 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.294906    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q"]
Oct 31 14:00:00.295080 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.294995    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q"
Oct 31 14:00:00.302642 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.301101    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp"]
Oct 31 14:00:00.307479 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.307067    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp"
Oct 31 14:00:00.314345 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.314212    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66"]
Oct 31 14:00:00.319034 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.319013    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp"]
Oct 31 14:00:00.319138 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.319101    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66"
Oct 31 14:00:00.331453 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.330582    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66"]
Oct 31 14:00:00.359493 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.359451    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz8gt\" (UniqueName: \"kubernetes.io/projected/f0e18350-e8e5-4d56-89e9-dc8dbdf12de8-kube-api-access-tz8gt\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp\" (UID: \"f0e18350-e8e5-4d56-89e9-dc8dbdf12de8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp"
Oct 31 14:00:00.359705 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.359502    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8wp5\" (UniqueName: \"kubernetes.io/projected/05b7941b-bb75-4e52-8766-7f7eb2f96734-kube-api-access-w8wp5\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp\" (UID: \"05b7941b-bb75-4e52-8766-7f7eb2f96734\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp"
Oct 31 14:00:00.359705 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.359596    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fee8a48-bc4e-4df9-b4db-354d2f577fd0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n\" (UID: \"0fee8a48-bc4e-4df9-b4db-354d2f577fd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n"
Oct 31 14:00:00.359705 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.359653    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxlrx\" (UniqueName: \"kubernetes.io/projected/b49565e4-99d7-4a99-a912-047c19c4b8b6-kube-api-access-zxlrx\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82\" (UID: \"b49565e4-99d7-4a99-a912-047c19c4b8b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82"
Oct 31 14:00:00.359905 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.359833    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49565e4-99d7-4a99-a912-047c19c4b8b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82\" (UID: \"b49565e4-99d7-4a99-a912-047c19c4b8b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82"
Oct 31 14:00:00.359965 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.359949    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab8637d-ccad-4855-b7ff-455c744f3da5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q\" (UID: \"9ab8637d-ccad-4855-b7ff-455c744f3da5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q"
Oct 31 14:00:00.360032 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.359979    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sqhz\" (UniqueName: \"kubernetes.io/projected/9ab8637d-ccad-4855-b7ff-455c744f3da5-kube-api-access-5sqhz\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q\" (UID: \"9ab8637d-ccad-4855-b7ff-455c744f3da5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q"
Oct 31 14:00:00.360103 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.360043    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e18350-e8e5-4d56-89e9-dc8dbdf12de8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp\" (UID: \"f0e18350-e8e5-4d56-89e9-dc8dbdf12de8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp"
Oct 31 14:00:00.360103 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.360083    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/124e886b-210d-465f-b6a2-5dcd03b03457-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg\" (UID: \"124e886b-210d-465f-b6a2-5dcd03b03457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg"
Oct 31 14:00:00.360227 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.360162    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hcfc\" (UniqueName: \"kubernetes.io/projected/124e886b-210d-465f-b6a2-5dcd03b03457-kube-api-access-2hcfc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg\" (UID: \"124e886b-210d-465f-b6a2-5dcd03b03457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg"
Oct 31 14:00:00.360227 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.360210    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7rjf\" (UniqueName: \"kubernetes.io/projected/0fee8a48-bc4e-4df9-b4db-354d2f577fd0-kube-api-access-f7rjf\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n\" (UID: \"0fee8a48-bc4e-4df9-b4db-354d2f577fd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n"
Oct 31 14:00:00.360342 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.360275    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fee8a48-bc4e-4df9-b4db-354d2f577fd0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n\" (UID: \"0fee8a48-bc4e-4df9-b4db-354d2f577fd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n"
Oct 31 14:00:00.360342 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.360281    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05b7941b-bb75-4e52-8766-7f7eb2f96734-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp\" (UID: \"05b7941b-bb75-4e52-8766-7f7eb2f96734\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp"
Oct 31 14:00:00.360457 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.360381    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7pj5\" (UniqueName: \"kubernetes.io/projected/d7298101-5b09-4055-964b-1457aeaa8c0f-kube-api-access-g7pj5\") pod \"maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66\" (UID: \"d7298101-5b09-4055-964b-1457aeaa8c0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66"
Oct 31 14:00:00.360457 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.360414    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7298101-5b09-4055-964b-1457aeaa8c0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66\" (UID: \"d7298101-5b09-4055-964b-1457aeaa8c0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66"
Oct 31 14:00:00.360582 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.360561    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49565e4-99d7-4a99-a912-047c19c4b8b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82\" (UID: \"b49565e4-99d7-4a99-a912-047c19c4b8b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82"
Oct 31 14:00:00.360719 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.360701    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/124e886b-210d-465f-b6a2-5dcd03b03457-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg\" (UID: \"124e886b-210d-465f-b6a2-5dcd03b03457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg"
Oct 31 14:00:00.372997 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.372969    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hcfc\" (UniqueName: \"kubernetes.io/projected/124e886b-210d-465f-b6a2-5dcd03b03457-kube-api-access-2hcfc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg\" (UID: \"124e886b-210d-465f-b6a2-5dcd03b03457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg"
Oct 31 14:00:00.373130 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.372982    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxlrx\" (UniqueName: \"kubernetes.io/projected/b49565e4-99d7-4a99-a912-047c19c4b8b6-kube-api-access-zxlrx\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82\" (UID: \"b49565e4-99d7-4a99-a912-047c19c4b8b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82"
Oct 31 14:00:00.373178 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.373143    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7rjf\" (UniqueName: \"kubernetes.io/projected/0fee8a48-bc4e-4df9-b4db-354d2f577fd0-kube-api-access-f7rjf\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n\" (UID: \"0fee8a48-bc4e-4df9-b4db-354d2f577fd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n"
Oct 31 14:00:00.462082 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.462055    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05b7941b-bb75-4e52-8766-7f7eb2f96734-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp\" (UID: \"05b7941b-bb75-4e52-8766-7f7eb2f96734\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp"
Oct 31 14:00:00.462251 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.462119    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7pj5\" (UniqueName: \"kubernetes.io/projected/d7298101-5b09-4055-964b-1457aeaa8c0f-kube-api-access-g7pj5\") pod \"maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66\" (UID: \"d7298101-5b09-4055-964b-1457aeaa8c0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66"
Oct 31 14:00:00.462251 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.462138    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7298101-5b09-4055-964b-1457aeaa8c0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66\" (UID: \"d7298101-5b09-4055-964b-1457aeaa8c0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66"
Oct 31 14:00:00.462251 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.462170    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz8gt\" (UniqueName: \"kubernetes.io/projected/f0e18350-e8e5-4d56-89e9-dc8dbdf12de8-kube-api-access-tz8gt\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp\" (UID: \"f0e18350-e8e5-4d56-89e9-dc8dbdf12de8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp"
Oct 31 14:00:00.462251 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.462194    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8wp5\" (UniqueName: \"kubernetes.io/projected/05b7941b-bb75-4e52-8766-7f7eb2f96734-kube-api-access-w8wp5\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp\" (UID: \"05b7941b-bb75-4e52-8766-7f7eb2f96734\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp"
Oct 31 14:00:00.462509 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.462415    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab8637d-ccad-4855-b7ff-455c744f3da5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q\" (UID: \"9ab8637d-ccad-4855-b7ff-455c744f3da5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q"
Oct 31 14:00:00.462509 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.462449    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sqhz\" (UniqueName: \"kubernetes.io/projected/9ab8637d-ccad-4855-b7ff-455c744f3da5-kube-api-access-5sqhz\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q\" (UID: \"9ab8637d-ccad-4855-b7ff-455c744f3da5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q"
Oct 31 14:00:00.462509 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.462479    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e18350-e8e5-4d56-89e9-dc8dbdf12de8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp\" (UID: \"f0e18350-e8e5-4d56-89e9-dc8dbdf12de8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp"
Oct 31 14:00:00.462936 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.462906    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05b7941b-bb75-4e52-8766-7f7eb2f96734-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp\" (UID: \"05b7941b-bb75-4e52-8766-7f7eb2f96734\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp"
Oct 31 14:00:00.463078 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.463051    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7298101-5b09-4055-964b-1457aeaa8c0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66\" (UID: \"d7298101-5b09-4055-964b-1457aeaa8c0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66"
Oct 31 14:00:00.463176 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.463155    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab8637d-ccad-4855-b7ff-455c744f3da5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q\" (UID: \"9ab8637d-ccad-4855-b7ff-455c744f3da5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q"
Oct 31 14:00:00.463240 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.463209    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e18350-e8e5-4d56-89e9-dc8dbdf12de8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp\" (UID: \"f0e18350-e8e5-4d56-89e9-dc8dbdf12de8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp"
Oct 31 14:00:00.476477 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.476445    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7pj5\" (UniqueName: \"kubernetes.io/projected/d7298101-5b09-4055-964b-1457aeaa8c0f-kube-api-access-g7pj5\") pod \"maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66\" (UID: \"d7298101-5b09-4055-964b-1457aeaa8c0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66"
Oct 31 14:00:00.476638 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.476449    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sqhz\" (UniqueName: \"kubernetes.io/projected/9ab8637d-ccad-4855-b7ff-455c744f3da5-kube-api-access-5sqhz\") pod \"maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q\" (UID: \"9ab8637d-ccad-4855-b7ff-455c744f3da5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q"
Oct 31 14:00:00.476707 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.476636    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8wp5\" (UniqueName: \"kubernetes.io/projected/05b7941b-bb75-4e52-8766-7f7eb2f96734-kube-api-access-w8wp5\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp\" (UID: \"05b7941b-bb75-4e52-8766-7f7eb2f96734\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp"
Oct 31 14:00:00.476707 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.476647    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz8gt\" (UniqueName: \"kubernetes.io/projected/f0e18350-e8e5-4d56-89e9-dc8dbdf12de8-kube-api-access-tz8gt\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp\" (UID: \"f0e18350-e8e5-4d56-89e9-dc8dbdf12de8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp"
Oct 31 14:00:00.482535 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.482476    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82"
Oct 31 14:00:00.549940 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.549721    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n"
Oct 31 14:00:00.594291 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.594254    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg"
Oct 31 14:00:00.601935 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.601106    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp"
Oct 31 14:00:00.611712 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.611475    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q"
Oct 31 14:00:00.628373 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.625706    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp"
Oct 31 14:00:00.646804 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.645525    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66"
Oct 31 14:00:00.794999 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.792483    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="509c1833-4923-4718-9661-1cf9a861fdbb" path="/var/lib/kubelet/pods/509c1833-4923-4718-9661-1cf9a861fdbb/volumes"
Oct 31 14:00:00.819821 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.819640    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:00.831115 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.831049    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82"]
Oct 31 14:00:00.943704 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:00.943677    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n"]
Oct 31 14:00:01.131505 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:01.131481    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg"]
Oct 31 14:00:01.135803 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:01.135778    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66"]
Oct 31 14:00:01.140047 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:01.140020    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp"]
Oct 31 14:00:01.157626 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:00:01.157595    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124e886b_210d_465f_b6a2_5dcd03b03457.slice/crio-afdf21f13f13af53f104d0a363edf24e6b44f968b86bf85890843f6cdc49e0ca WatchSource:0}: Error finding container afdf21f13f13af53f104d0a363edf24e6b44f968b86bf85890843f6cdc49e0ca: Status 404 returned error can't find the container with id afdf21f13f13af53f104d0a363edf24e6b44f968b86bf85890843f6cdc49e0ca
Oct 31 14:00:01.158098 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:00:01.158063    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7298101_5b09_4055_964b_1457aeaa8c0f.slice/crio-30a9ddf2d35fb2fd4cca65b650779f886a5ef4094b3e5914d8595e4d39b784f9 WatchSource:0}: Error finding container 30a9ddf2d35fb2fd4cca65b650779f886a5ef4094b3e5914d8595e4d39b784f9: Status 404 returned error can't find the container with id 30a9ddf2d35fb2fd4cca65b650779f886a5ef4094b3e5914d8595e4d39b784f9
Oct 31 14:00:01.158325 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:00:01.158299    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b7941b_bb75_4e52_8766_7f7eb2f96734.slice/crio-6b7ec85ce3418c84c941dda438278f7f5382f6c88fd7533f502fb2ef0538913f WatchSource:0}: Error finding container 6b7ec85ce3418c84c941dda438278f7f5382f6c88fd7533f502fb2ef0538913f: Status 404 returned error can't find the container with id 6b7ec85ce3418c84c941dda438278f7f5382f6c88fd7533f502fb2ef0538913f
Oct 31 14:00:01.159293 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:01.159256    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n" event={"ID":"0fee8a48-bc4e-4df9-b4db-354d2f577fd0","Type":"ContainerStarted","Data":"6556d7390c82a921ed905ce55fc1a44fae8c8f573f089a06bc650a08780a984a"}
Oct 31 14:00:01.163356 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:01.163335    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82" event={"ID":"b49565e4-99d7-4a99-a912-047c19c4b8b6","Type":"ContainerStarted","Data":"b2e9b589ba6d450faf0c5fed7adab81729db687f6a802aff02be7e39b52476b0"}
Oct 31 14:00:01.214829 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:01.214800    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q"]
Oct 31 14:00:01.215172 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:00:01.215105    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab8637d_ccad_4855_b7ff_455c744f3da5.slice/crio-19fe0381e3525c22168d52d9c2afb637456568bd58428b40ddd4ba957594b4ab WatchSource:0}: Error finding container 19fe0381e3525c22168d52d9c2afb637456568bd58428b40ddd4ba957594b4ab: Status 404 returned error can't find the container with id 19fe0381e3525c22168d52d9c2afb637456568bd58428b40ddd4ba957594b4ab
Oct 31 14:00:01.218248 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:00:01.218218    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e18350_e8e5_4d56_89e9_dc8dbdf12de8.slice/crio-2b590f875cf37e2b5bb7cb4616626794726f83f2812a31dd1cd49fb596b814a7 WatchSource:0}: Error finding container 2b590f875cf37e2b5bb7cb4616626794726f83f2812a31dd1cd49fb596b814a7: Status 404 returned error can't find the container with id 2b590f875cf37e2b5bb7cb4616626794726f83f2812a31dd1cd49fb596b814a7
Oct 31 14:00:01.220014 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:01.219988    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp"]
Oct 31 14:00:01.844554 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:01.844392    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:02.172778 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.172665    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp" event={"ID":"05b7941b-bb75-4e52-8766-7f7eb2f96734","Type":"ContainerStarted","Data":"bf9910c977e22001efd21163a4a35e966bc87e99335f7a563e54335e2dea8025"}
Oct 31 14:00:02.172778 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.172736    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp" event={"ID":"05b7941b-bb75-4e52-8766-7f7eb2f96734","Type":"ContainerStarted","Data":"6b7ec85ce3418c84c941dda438278f7f5382f6c88fd7533f502fb2ef0538913f"}
Oct 31 14:00:02.174560 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.174535    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66" event={"ID":"d7298101-5b09-4055-964b-1457aeaa8c0f","Type":"ContainerStarted","Data":"2f319630a6ddbe973c1edcce2a8dbd86581ab7fef4d594ccb08c5a772532fc6a"}
Oct 31 14:00:02.174708 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.174586    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66" event={"ID":"d7298101-5b09-4055-964b-1457aeaa8c0f","Type":"ContainerStarted","Data":"30a9ddf2d35fb2fd4cca65b650779f886a5ef4094b3e5914d8595e4d39b784f9"}
Oct 31 14:00:02.176230 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.176208    4428 generic.go:356] "Generic (PLEG): container finished" podID="b49565e4-99d7-4a99-a912-047c19c4b8b6" containerID="d2ae7dfbe7116c008dec6e29cdaa9c50908eaa4ef643caab195f04541e3cdafc" exitCode=60
Oct 31 14:00:02.176342 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.176291    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82" event={"ID":"b49565e4-99d7-4a99-a912-047c19c4b8b6","Type":"ContainerDied","Data":"d2ae7dfbe7116c008dec6e29cdaa9c50908eaa4ef643caab195f04541e3cdafc"}
Oct 31 14:00:02.176586 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.176557    4428 scope.go:122] "RemoveContainer" containerID="d2ae7dfbe7116c008dec6e29cdaa9c50908eaa4ef643caab195f04541e3cdafc"
Oct 31 14:00:02.178605 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.178572    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q" event={"ID":"9ab8637d-ccad-4855-b7ff-455c744f3da5","Type":"ContainerStarted","Data":"832055294aef6631bc17ae08f30a24dae264c420ad506e55410b4da93898eba0"}
Oct 31 14:00:02.178710 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.178667    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q" event={"ID":"9ab8637d-ccad-4855-b7ff-455c744f3da5","Type":"ContainerStarted","Data":"19fe0381e3525c22168d52d9c2afb637456568bd58428b40ddd4ba957594b4ab"}
Oct 31 14:00:02.188270 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.188230    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n" event={"ID":"0fee8a48-bc4e-4df9-b4db-354d2f577fd0","Type":"ContainerStarted","Data":"90b4ae8617bff6c5c5b8d3e0944e34c286060f54567f38e59be0566f4e7758a1"}
Oct 31 14:00:02.192837 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.192799    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp" event={"ID":"f0e18350-e8e5-4d56-89e9-dc8dbdf12de8","Type":"ContainerStarted","Data":"f7512525ccbbbafc340ddfd6946a6bf21d4fd3fe2523d7643536ca2933af3d9d"}
Oct 31 14:00:02.192986 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.192966    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp" event={"ID":"f0e18350-e8e5-4d56-89e9-dc8dbdf12de8","Type":"ContainerStarted","Data":"2b590f875cf37e2b5bb7cb4616626794726f83f2812a31dd1cd49fb596b814a7"}
Oct 31 14:00:02.195667 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.195636    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg" event={"ID":"124e886b-210d-465f-b6a2-5dcd03b03457","Type":"ContainerStarted","Data":"5471b42ab233f6fc1a0fa69aafdabd892e63b20c9b8ca9990598a745fa5b1440"}
Oct 31 14:00:02.195791 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.195694    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg" event={"ID":"124e886b-210d-465f-b6a2-5dcd03b03457","Type":"ContainerStarted","Data":"afdf21f13f13af53f104d0a363edf24e6b44f968b86bf85890843f6cdc49e0ca"}
Oct 31 14:00:02.200998 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.200921    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp" podStartSLOduration=2.200846686 podStartE2EDuration="2.200846686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:02.192481135 +0000 UTC m=+3399.590829759" watchObservedRunningTime="2026-10-31 14:00:02.200846686 +0000 UTC m=+3399.599195311"
Oct 31 14:00:02.214701 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.214639    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q" podStartSLOduration=2.2146189290000002 podStartE2EDuration="2.214618929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:02.210781662 +0000 UTC m=+3399.609130296" watchObservedRunningTime="2026-10-31 14:00:02.214618929 +0000 UTC m=+3399.612967550"
Oct 31 14:00:02.230582 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.230530    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66" podStartSLOduration=2.230514934 podStartE2EDuration="2.230514934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:02.228663397 +0000 UTC m=+3399.627012056" watchObservedRunningTime="2026-10-31 14:00:02.230514934 +0000 UTC m=+3399.628863559"
Oct 31 14:00:02.253282 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.253235    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n" podStartSLOduration=2.253220062 podStartE2EDuration="2.253220062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:02.24774194 +0000 UTC m=+3399.646090565" watchObservedRunningTime="2026-10-31 14:00:02.253220062 +0000 UTC m=+3399.651568686"
Oct 31 14:00:02.288854 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.288792    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg" podStartSLOduration=2.288777959 podStartE2EDuration="2.288777959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:02.287296834 +0000 UTC m=+3399.685645460" watchObservedRunningTime="2026-10-31 14:00:02.288777959 +0000 UTC m=+3399.687126582"
Oct 31 14:00:02.307666 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.307599    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp" podStartSLOduration=2.307578672 podStartE2EDuration="2.307578672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:02.305948652 +0000 UTC m=+3399.704297277" watchObservedRunningTime="2026-10-31 14:00:02.307578672 +0000 UTC m=+3399.705927297"
Oct 31 14:00:02.723183 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:02.722014    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:03.212093 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.212050    4428 generic.go:356] "Generic (PLEG): container finished" podID="d7298101-5b09-4055-964b-1457aeaa8c0f" containerID="2f319630a6ddbe973c1edcce2a8dbd86581ab7fef4d594ccb08c5a772532fc6a" exitCode=60
Oct 31 14:00:03.212575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.212140    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66" event={"ID":"d7298101-5b09-4055-964b-1457aeaa8c0f","Type":"ContainerDied","Data":"2f319630a6ddbe973c1edcce2a8dbd86581ab7fef4d594ccb08c5a772532fc6a"}
Oct 31 14:00:03.212575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.212492    4428 scope.go:122] "RemoveContainer" containerID="2f319630a6ddbe973c1edcce2a8dbd86581ab7fef4d594ccb08c5a772532fc6a"
Oct 31 14:00:03.214581 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.214452    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82" event={"ID":"b49565e4-99d7-4a99-a912-047c19c4b8b6","Type":"ContainerStarted","Data":"c3add6cf8867c509a378611c3373777e59ebcabed697631f38978b1e87cf293f"}
Oct 31 14:00:03.216236 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.216202    4428 generic.go:356] "Generic (PLEG): container finished" podID="9ab8637d-ccad-4855-b7ff-455c744f3da5" containerID="832055294aef6631bc17ae08f30a24dae264c420ad506e55410b4da93898eba0" exitCode=60
Oct 31 14:00:03.216370 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.216280    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q" event={"ID":"9ab8637d-ccad-4855-b7ff-455c744f3da5","Type":"ContainerDied","Data":"832055294aef6631bc17ae08f30a24dae264c420ad506e55410b4da93898eba0"}
Oct 31 14:00:03.216693 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.216621    4428 scope.go:122] "RemoveContainer" containerID="832055294aef6631bc17ae08f30a24dae264c420ad506e55410b4da93898eba0"
Oct 31 14:00:03.221435 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.221414    4428 generic.go:356] "Generic (PLEG): container finished" podID="0fee8a48-bc4e-4df9-b4db-354d2f577fd0" containerID="90b4ae8617bff6c5c5b8d3e0944e34c286060f54567f38e59be0566f4e7758a1" exitCode=60
Oct 31 14:00:03.221527 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.221494    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n" event={"ID":"0fee8a48-bc4e-4df9-b4db-354d2f577fd0","Type":"ContainerDied","Data":"90b4ae8617bff6c5c5b8d3e0944e34c286060f54567f38e59be0566f4e7758a1"}
Oct 31 14:00:03.221796 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.221781    4428 scope.go:122] "RemoveContainer" containerID="90b4ae8617bff6c5c5b8d3e0944e34c286060f54567f38e59be0566f4e7758a1"
Oct 31 14:00:03.223569 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.223145    4428 generic.go:356] "Generic (PLEG): container finished" podID="f0e18350-e8e5-4d56-89e9-dc8dbdf12de8" containerID="f7512525ccbbbafc340ddfd6946a6bf21d4fd3fe2523d7643536ca2933af3d9d" exitCode=60
Oct 31 14:00:03.223569 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.223243    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp" event={"ID":"f0e18350-e8e5-4d56-89e9-dc8dbdf12de8","Type":"ContainerDied","Data":"f7512525ccbbbafc340ddfd6946a6bf21d4fd3fe2523d7643536ca2933af3d9d"}
Oct 31 14:00:03.224550 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.224532    4428 scope.go:122] "RemoveContainer" containerID="f7512525ccbbbafc340ddfd6946a6bf21d4fd3fe2523d7643536ca2933af3d9d"
Oct 31 14:00:03.225457 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.225438    4428 generic.go:356] "Generic (PLEG): container finished" podID="124e886b-210d-465f-b6a2-5dcd03b03457" containerID="5471b42ab233f6fc1a0fa69aafdabd892e63b20c9b8ca9990598a745fa5b1440" exitCode=60
Oct 31 14:00:03.225541 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.225520    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg" event={"ID":"124e886b-210d-465f-b6a2-5dcd03b03457","Type":"ContainerDied","Data":"5471b42ab233f6fc1a0fa69aafdabd892e63b20c9b8ca9990598a745fa5b1440"}
Oct 31 14:00:03.226143 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.226122    4428 scope.go:122] "RemoveContainer" containerID="5471b42ab233f6fc1a0fa69aafdabd892e63b20c9b8ca9990598a745fa5b1440"
Oct 31 14:00:03.227298 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.227272    4428 generic.go:356] "Generic (PLEG): container finished" podID="05b7941b-bb75-4e52-8766-7f7eb2f96734" containerID="bf9910c977e22001efd21163a4a35e966bc87e99335f7a563e54335e2dea8025" exitCode=60
Oct 31 14:00:03.227394 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.227331    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp" event={"ID":"05b7941b-bb75-4e52-8766-7f7eb2f96734","Type":"ContainerDied","Data":"bf9910c977e22001efd21163a4a35e966bc87e99335f7a563e54335e2dea8025"}
Oct 31 14:00:03.227623 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.227604    4428 scope.go:122] "RemoveContainer" containerID="bf9910c977e22001efd21163a4a35e966bc87e99335f7a563e54335e2dea8025"
Oct 31 14:00:03.281068 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.280252    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82" podStartSLOduration=3.280231541 podStartE2EDuration="3.280231541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:03.278679449 +0000 UTC m=+3400.677028077" watchObservedRunningTime="2026-10-31 14:00:03.280231541 +0000 UTC m=+3400.678580167"
Oct 31 14:00:03.776895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:03.768151    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:04.237662 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:04.237621    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp" event={"ID":"05b7941b-bb75-4e52-8766-7f7eb2f96734","Type":"ContainerStarted","Data":"a8a160ee900877575e32f4c337e2090822afefb6bb76443f8660ad9d114578a3"}
Oct 31 14:00:04.239909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:04.239844    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66" event={"ID":"d7298101-5b09-4055-964b-1457aeaa8c0f","Type":"ContainerStarted","Data":"dd475ac556e44bc16b0cd95594dc3c43d073a9e5f2f7d9cfb6fdc243f79c3647"}
Oct 31 14:00:04.242009 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:04.241972    4428 generic.go:356] "Generic (PLEG): container finished" podID="b49565e4-99d7-4a99-a912-047c19c4b8b6" containerID="c3add6cf8867c509a378611c3373777e59ebcabed697631f38978b1e87cf293f" exitCode=60
Oct 31 14:00:04.242144 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:04.242066    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82" event={"ID":"b49565e4-99d7-4a99-a912-047c19c4b8b6","Type":"ContainerDied","Data":"c3add6cf8867c509a378611c3373777e59ebcabed697631f38978b1e87cf293f"}
Oct 31 14:00:04.242192 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:04.242146    4428 scope.go:122] "RemoveContainer" containerID="d2ae7dfbe7116c008dec6e29cdaa9c50908eaa4ef643caab195f04541e3cdafc"
Oct 31 14:00:04.242322 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:04.242308    4428 scope.go:122] "RemoveContainer" containerID="c3add6cf8867c509a378611c3373777e59ebcabed697631f38978b1e87cf293f"
Oct 31 14:00:04.242668 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:04.242642    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82_odh-ai-gateway-infra(b49565e4-99d7-4a99-a912-047c19c4b8b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82" podUID="b49565e4-99d7-4a99-a912-047c19c4b8b6"
Oct 31 14:00:04.244768 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:04.244745    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q" event={"ID":"9ab8637d-ccad-4855-b7ff-455c744f3da5","Type":"ContainerStarted","Data":"5dd3c77c1bab081af45ef63a477e39bab88db81a753b0fa007f731d33b5d554d"}
Oct 31 14:00:04.248763 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:04.248737    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n" event={"ID":"0fee8a48-bc4e-4df9-b4db-354d2f577fd0","Type":"ContainerStarted","Data":"e4a57f40a97c1350a0f761a616d54a95492dc170c98db044d9679231188019d4"}
Oct 31 14:00:04.253062 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:04.253029    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp" event={"ID":"f0e18350-e8e5-4d56-89e9-dc8dbdf12de8","Type":"ContainerStarted","Data":"e5cc63e1d4833c44ad12ef2600718628d6d87df353e47b9163674b0e99cdae8a"}
Oct 31 14:00:04.255222 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:04.255195    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg" event={"ID":"124e886b-210d-465f-b6a2-5dcd03b03457","Type":"ContainerStarted","Data":"10e6e30801163e636c73dd52ff6dc04dd9b55024294bc3e692f80e5bc5ba8995"}
Oct 31 14:00:04.694553 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:04.694466    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:05.265350 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.265311    4428 scope.go:122] "RemoveContainer" containerID="c3add6cf8867c509a378611c3373777e59ebcabed697631f38978b1e87cf293f"
Oct 31 14:00:05.265844 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:05.265714    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82_odh-ai-gateway-infra(b49565e4-99d7-4a99-a912-047c19c4b8b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82" podUID="b49565e4-99d7-4a99-a912-047c19c4b8b6"
Oct 31 14:00:05.267504 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.267437    4428 generic.go:356] "Generic (PLEG): container finished" podID="9ab8637d-ccad-4855-b7ff-455c744f3da5" containerID="5dd3c77c1bab081af45ef63a477e39bab88db81a753b0fa007f731d33b5d554d" exitCode=60
Oct 31 14:00:05.267834 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.267541    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q" event={"ID":"9ab8637d-ccad-4855-b7ff-455c744f3da5","Type":"ContainerDied","Data":"5dd3c77c1bab081af45ef63a477e39bab88db81a753b0fa007f731d33b5d554d"}
Oct 31 14:00:05.267834 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.267653    4428 scope.go:122] "RemoveContainer" containerID="832055294aef6631bc17ae08f30a24dae264c420ad506e55410b4da93898eba0"
Oct 31 14:00:05.268310 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.268277    4428 scope.go:122] "RemoveContainer" containerID="5dd3c77c1bab081af45ef63a477e39bab88db81a753b0fa007f731d33b5d554d"
Oct 31 14:00:05.269019 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:05.268971    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q_odh-ai-gateway-infra(9ab8637d-ccad-4855-b7ff-455c744f3da5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q" podUID="9ab8637d-ccad-4855-b7ff-455c744f3da5"
Oct 31 14:00:05.271427 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.271394    4428 generic.go:356] "Generic (PLEG): container finished" podID="0fee8a48-bc4e-4df9-b4db-354d2f577fd0" containerID="e4a57f40a97c1350a0f761a616d54a95492dc170c98db044d9679231188019d4" exitCode=60
Oct 31 14:00:05.271601 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.271502    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n" event={"ID":"0fee8a48-bc4e-4df9-b4db-354d2f577fd0","Type":"ContainerDied","Data":"e4a57f40a97c1350a0f761a616d54a95492dc170c98db044d9679231188019d4"}
Oct 31 14:00:05.271946 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.271924    4428 scope.go:122] "RemoveContainer" containerID="e4a57f40a97c1350a0f761a616d54a95492dc170c98db044d9679231188019d4"
Oct 31 14:00:05.272455 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:05.272415    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n_odh-ai-gateway-infra(0fee8a48-bc4e-4df9-b4db-354d2f577fd0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n" podUID="0fee8a48-bc4e-4df9-b4db-354d2f577fd0"
Oct 31 14:00:05.274048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.274016    4428 generic.go:356] "Generic (PLEG): container finished" podID="f0e18350-e8e5-4d56-89e9-dc8dbdf12de8" containerID="e5cc63e1d4833c44ad12ef2600718628d6d87df353e47b9163674b0e99cdae8a" exitCode=60
Oct 31 14:00:05.274171 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.274104    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp" event={"ID":"f0e18350-e8e5-4d56-89e9-dc8dbdf12de8","Type":"ContainerDied","Data":"e5cc63e1d4833c44ad12ef2600718628d6d87df353e47b9163674b0e99cdae8a"}
Oct 31 14:00:05.274422 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.274398    4428 scope.go:122] "RemoveContainer" containerID="e5cc63e1d4833c44ad12ef2600718628d6d87df353e47b9163674b0e99cdae8a"
Oct 31 14:00:05.274766 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:05.274748    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp_odh-ai-gateway-infra(f0e18350-e8e5-4d56-89e9-dc8dbdf12de8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp" podUID="f0e18350-e8e5-4d56-89e9-dc8dbdf12de8"
Oct 31 14:00:05.276628 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.276602    4428 generic.go:356] "Generic (PLEG): container finished" podID="124e886b-210d-465f-b6a2-5dcd03b03457" containerID="10e6e30801163e636c73dd52ff6dc04dd9b55024294bc3e692f80e5bc5ba8995" exitCode=60
Oct 31 14:00:05.276737 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.276677    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg" event={"ID":"124e886b-210d-465f-b6a2-5dcd03b03457","Type":"ContainerDied","Data":"10e6e30801163e636c73dd52ff6dc04dd9b55024294bc3e692f80e5bc5ba8995"}
Oct 31 14:00:05.277020 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.277000    4428 scope.go:122] "RemoveContainer" containerID="10e6e30801163e636c73dd52ff6dc04dd9b55024294bc3e692f80e5bc5ba8995"
Oct 31 14:00:05.277414 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:05.277388    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg_odh-ai-gateway-infra(124e886b-210d-465f-b6a2-5dcd03b03457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg" podUID="124e886b-210d-465f-b6a2-5dcd03b03457"
Oct 31 14:00:05.279106 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.279086    4428 generic.go:356] "Generic (PLEG): container finished" podID="05b7941b-bb75-4e52-8766-7f7eb2f96734" containerID="a8a160ee900877575e32f4c337e2090822afefb6bb76443f8660ad9d114578a3" exitCode=60
Oct 31 14:00:05.279214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.279175    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp" event={"ID":"05b7941b-bb75-4e52-8766-7f7eb2f96734","Type":"ContainerDied","Data":"a8a160ee900877575e32f4c337e2090822afefb6bb76443f8660ad9d114578a3"}
Oct 31 14:00:05.279445 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.279430    4428 scope.go:122] "RemoveContainer" containerID="a8a160ee900877575e32f4c337e2090822afefb6bb76443f8660ad9d114578a3"
Oct 31 14:00:05.279777 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:05.279757    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp_odh-ai-gateway-infra(05b7941b-bb75-4e52-8766-7f7eb2f96734)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp" podUID="05b7941b-bb75-4e52-8766-7f7eb2f96734"
Oct 31 14:00:05.281438 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.281415    4428 generic.go:356] "Generic (PLEG): container finished" podID="d7298101-5b09-4055-964b-1457aeaa8c0f" containerID="dd475ac556e44bc16b0cd95594dc3c43d073a9e5f2f7d9cfb6fdc243f79c3647" exitCode=60
Oct 31 14:00:05.281552 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.281450    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66" event={"ID":"d7298101-5b09-4055-964b-1457aeaa8c0f","Type":"ContainerDied","Data":"dd475ac556e44bc16b0cd95594dc3c43d073a9e5f2f7d9cfb6fdc243f79c3647"}
Oct 31 14:00:05.281730 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.281714    4428 scope.go:122] "RemoveContainer" containerID="dd475ac556e44bc16b0cd95594dc3c43d073a9e5f2f7d9cfb6fdc243f79c3647"
Oct 31 14:00:05.282065 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:05.282042    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66_odh-ai-gateway-infra(d7298101-5b09-4055-964b-1457aeaa8c0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66" podUID="d7298101-5b09-4055-964b-1457aeaa8c0f"
Oct 31 14:00:05.285334 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.285309    4428 scope.go:122] "RemoveContainer" containerID="90b4ae8617bff6c5c5b8d3e0944e34c286060f54567f38e59be0566f4e7758a1"
Oct 31 14:00:05.300070 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.300039    4428 scope.go:122] "RemoveContainer" containerID="f7512525ccbbbafc340ddfd6946a6bf21d4fd3fe2523d7643536ca2933af3d9d"
Oct 31 14:00:05.321666 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.321637    4428 scope.go:122] "RemoveContainer" containerID="5471b42ab233f6fc1a0fa69aafdabd892e63b20c9b8ca9990598a745fa5b1440"
Oct 31 14:00:05.338289 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.338263    4428 scope.go:122] "RemoveContainer" containerID="bf9910c977e22001efd21163a4a35e966bc87e99335f7a563e54335e2dea8025"
Oct 31 14:00:05.353186 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.353160    4428 scope.go:122] "RemoveContainer" containerID="2f319630a6ddbe973c1edcce2a8dbd86581ab7fef4d594ccb08c5a772532fc6a"
Oct 31 14:00:05.618949 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:05.604845    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:06.292292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:06.292263    4428 scope.go:122] "RemoveContainer" containerID="dd475ac556e44bc16b0cd95594dc3c43d073a9e5f2f7d9cfb6fdc243f79c3647"
Oct 31 14:00:06.292738 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:06.292719    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66_odh-ai-gateway-infra(d7298101-5b09-4055-964b-1457aeaa8c0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66" podUID="d7298101-5b09-4055-964b-1457aeaa8c0f"
Oct 31 14:00:06.294112 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:06.294093    4428 scope.go:122] "RemoveContainer" containerID="5dd3c77c1bab081af45ef63a477e39bab88db81a753b0fa007f731d33b5d554d"
Oct 31 14:00:06.294444 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:06.294422    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q_odh-ai-gateway-infra(9ab8637d-ccad-4855-b7ff-455c744f3da5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q" podUID="9ab8637d-ccad-4855-b7ff-455c744f3da5"
Oct 31 14:00:06.295813 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:06.295792    4428 scope.go:122] "RemoveContainer" containerID="e4a57f40a97c1350a0f761a616d54a95492dc170c98db044d9679231188019d4"
Oct 31 14:00:06.296178 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:06.296155    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n_odh-ai-gateway-infra(0fee8a48-bc4e-4df9-b4db-354d2f577fd0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n" podUID="0fee8a48-bc4e-4df9-b4db-354d2f577fd0"
Oct 31 14:00:06.297579 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:06.297558    4428 scope.go:122] "RemoveContainer" containerID="e5cc63e1d4833c44ad12ef2600718628d6d87df353e47b9163674b0e99cdae8a"
Oct 31 14:00:06.297965 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:06.297939    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp_odh-ai-gateway-infra(f0e18350-e8e5-4d56-89e9-dc8dbdf12de8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp" podUID="f0e18350-e8e5-4d56-89e9-dc8dbdf12de8"
Oct 31 14:00:06.299713 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:06.299691    4428 scope.go:122] "RemoveContainer" containerID="10e6e30801163e636c73dd52ff6dc04dd9b55024294bc3e692f80e5bc5ba8995"
Oct 31 14:00:06.300081 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:06.300044    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg_odh-ai-gateway-infra(124e886b-210d-465f-b6a2-5dcd03b03457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg" podUID="124e886b-210d-465f-b6a2-5dcd03b03457"
Oct 31 14:00:06.301378 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:06.301347    4428 scope.go:122] "RemoveContainer" containerID="a8a160ee900877575e32f4c337e2090822afefb6bb76443f8660ad9d114578a3"
Oct 31 14:00:06.301691 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:06.301668    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp_odh-ai-gateway-infra(05b7941b-bb75-4e52-8766-7f7eb2f96734)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp" podUID="05b7941b-bb75-4e52-8766-7f7eb2f96734"
Oct 31 14:00:06.625955 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:06.614384    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:07.163909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:07.163851    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:07.299140 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:07.299088    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"]
Oct 31 14:00:07.306946 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:07.306913    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"
Oct 31 14:00:07.318657 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:07.318622    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"]
Oct 31 14:00:07.371544 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:07.368269    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f9ca72\""
Oct 31 14:00:07.445953 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:07.380815    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:07.445953 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:07.381097    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f9ca72-dockercfg-j6qhd\""
Oct 31 14:00:07.445953 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:07.382373    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt98t\" (UniqueName: \"kubernetes.io/projected/2c55141e-2afe-4d0d-ba77-834eca70c27f-kube-api-access-tt98t\") pod \"payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc\" (UID: \"2c55141e-2afe-4d0d-ba77-834eca70c27f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"
Oct 31 14:00:07.445953 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:07.442255    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c55141e-2afe-4d0d-ba77-834eca70c27f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc\" (UID: \"2c55141e-2afe-4d0d-ba77-834eca70c27f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"
Oct 31 14:00:07.543921 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:07.543754    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt98t\" (UniqueName: \"kubernetes.io/projected/2c55141e-2afe-4d0d-ba77-834eca70c27f-kube-api-access-tt98t\") pod \"payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc\" (UID: \"2c55141e-2afe-4d0d-ba77-834eca70c27f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"
Oct 31 14:00:07.543921 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:07.543924    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c55141e-2afe-4d0d-ba77-834eca70c27f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc\" (UID: \"2c55141e-2afe-4d0d-ba77-834eca70c27f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"
Oct 31 14:00:07.545217 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:07.545176    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c55141e-2afe-4d0d-ba77-834eca70c27f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc\" (UID: \"2c55141e-2afe-4d0d-ba77-834eca70c27f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"
Oct 31 14:00:07.562853 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:07.562580    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt98t\" (UniqueName: \"kubernetes.io/projected/2c55141e-2afe-4d0d-ba77-834eca70c27f-kube-api-access-tt98t\") pod \"payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc\" (UID: \"2c55141e-2afe-4d0d-ba77-834eca70c27f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"
Oct 31 14:00:07.638112 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:07.638078    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"
Oct 31 14:00:07.924895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:07.915026    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:07.999045 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:07.999010    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"]
Oct 31 14:00:08.376985 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:08.376857    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc" event={"ID":"2c55141e-2afe-4d0d-ba77-834eca70c27f","Type":"ContainerStarted","Data":"b1f27fb7566dc8d9f06da33b34381501ceaf33178ac7c3a9985c7841f1569683"}
Oct 31 14:00:08.377573 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:08.377540    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc" event={"ID":"2c55141e-2afe-4d0d-ba77-834eca70c27f","Type":"ContainerStarted","Data":"1b47aefad773066867bb616c63eea5fd5d46462eff6285cb4f208b3ed722bad9"}
Oct 31 14:00:08.377701 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:08.377589    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"
Oct 31 14:00:08.400253 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:08.400191    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc" podStartSLOduration=1.400169119 podStartE2EDuration="1.400169119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:08.396728671 +0000 UTC m=+3405.795077295" watchObservedRunningTime="2026-10-31 14:00:08.400169119 +0000 UTC m=+3405.798517744"
Oct 31 14:00:08.938735 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:08.938703    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:09.873889 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:09.873849    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:10.624893 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:10.621655    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:11.463312 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:11.463246    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:12.313280 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:12.311407    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:13.109974 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:13.109915    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:13.748543 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:13.748447    4428 scope.go:122] "RemoveContainer" containerID="c3add6cf8867c509a378611c3373777e59ebcabed697631f38978b1e87cf293f"
Oct 31 14:00:13.851907 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:13.844709    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:14.423802 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.423712    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66"]
Oct 31 14:00:14.457849 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.457793    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82" event={"ID":"b49565e4-99d7-4a99-a912-047c19c4b8b6","Type":"ContainerStarted","Data":"f43ef616cbca816e300225cc893aaaa0a295e275485e64df6500d71958668ea9"}
Oct 31 14:00:14.492585 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.491649    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:14.501002 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.500964    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"]
Oct 31 14:00:14.501226 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.501200    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz" podUID="0e20d7a0-a20d-48fb-a666-c5c89ec7a927" containerName="payload-processing" containerID="cri-o://1b980d1002f58f4d44cb0ad875b7db658607cf113a782d703712ced96ca8e352" gracePeriod=30
Oct 31 14:00:14.510741 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.510693    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"]
Oct 31 14:00:14.512743 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.512713    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt" podUID="bacfb744-b683-4b4d-a4e8-eefeaa418d71" containerName="payload-pre-processing" containerID="cri-o://89e4713b3399c8f1fb2ab5be45a9db875aa234198c78679dbed3a2d9a1ee2200" gracePeriod=30
Oct 31 14:00:14.642914 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.642274    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"]
Oct 31 14:00:14.643109 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.642557    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq" podUID="5c65ac2b-9ff4-4752-8325-bb8bd027708f" containerName="maas-api" containerID="cri-o://e1fa52666d1ee9f296e94708571020058a39196843ac1e221bd1b594946fbd9f" gracePeriod=30
Oct 31 14:00:14.662049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.660893    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:14.735665 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.735631    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66"
Oct 31 14:00:14.783909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.782511    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7298101-5b09-4055-964b-1457aeaa8c0f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7298101-5b09-4055-964b-1457aeaa8c0f-maas-api-service-ca\") pod \"d7298101-5b09-4055-964b-1457aeaa8c0f\" (UID: \"d7298101-5b09-4055-964b-1457aeaa8c0f\") "
Oct 31 14:00:14.783909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.782593    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7298101-5b09-4055-964b-1457aeaa8c0f-kube-api-access-g7pj5\" (UniqueName: \"kubernetes.io/projected/d7298101-5b09-4055-964b-1457aeaa8c0f-kube-api-access-g7pj5\") pod \"d7298101-5b09-4055-964b-1457aeaa8c0f\" (UID: \"d7298101-5b09-4055-964b-1457aeaa8c0f\") "
Oct 31 14:00:14.785300 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.784954    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7298101-5b09-4055-964b-1457aeaa8c0f-maas-api-service-ca" pod "d7298101-5b09-4055-964b-1457aeaa8c0f" (UID: "d7298101-5b09-4055-964b-1457aeaa8c0f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:14.824548 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.824514    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:14.840245 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.840205    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7298101-5b09-4055-964b-1457aeaa8c0f-kube-api-access-g7pj5" pod "d7298101-5b09-4055-964b-1457aeaa8c0f" (UID: "d7298101-5b09-4055-964b-1457aeaa8c0f"). InnerVolumeSpecName "kube-api-access-g7pj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:14.886721 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.886684    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7298101-5b09-4055-964b-1457aeaa8c0f-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:14.886721 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.886721    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7pj5\" (UniqueName: \"kubernetes.io/projected/d7298101-5b09-4055-964b-1457aeaa8c0f-kube-api-access-g7pj5\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:14.989436 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:14.985878    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:15.211437 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.209655    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:15.241325 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.240163    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"
Oct 31 14:00:15.246159 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.246052    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"
Oct 31 14:00:15.314248 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.314221    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bacfb744-b683-4b4d-a4e8-eefeaa418d71-kube-api-access-wn9np\" (UniqueName: \"kubernetes.io/projected/bacfb744-b683-4b4d-a4e8-eefeaa418d71-kube-api-access-wn9np\") pod \"bacfb744-b683-4b4d-a4e8-eefeaa418d71\" (UID: \"bacfb744-b683-4b4d-a4e8-eefeaa418d71\") "
Oct 31 14:00:15.314496 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.314483    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bacfb744-b683-4b4d-a4e8-eefeaa418d71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacfb744-b683-4b4d-a4e8-eefeaa418d71-plugins-config-volume\") pod \"bacfb744-b683-4b4d-a4e8-eefeaa418d71\" (UID: \"bacfb744-b683-4b4d-a4e8-eefeaa418d71\") "
Oct 31 14:00:15.314703 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.314692    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e20d7a0-a20d-48fb-a666-c5c89ec7a927-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e20d7a0-a20d-48fb-a666-c5c89ec7a927-plugins-config-volume\") pod \"0e20d7a0-a20d-48fb-a666-c5c89ec7a927\" (UID: \"0e20d7a0-a20d-48fb-a666-c5c89ec7a927\") "
Oct 31 14:00:15.314787 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.314779    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e20d7a0-a20d-48fb-a666-c5c89ec7a927-kube-api-access-v5pqc\" (UniqueName: \"kubernetes.io/projected/0e20d7a0-a20d-48fb-a666-c5c89ec7a927-kube-api-access-v5pqc\") pod \"0e20d7a0-a20d-48fb-a666-c5c89ec7a927\" (UID: \"0e20d7a0-a20d-48fb-a666-c5c89ec7a927\") "
Oct 31 14:00:15.318897 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.318598    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bacfb744-b683-4b4d-a4e8-eefeaa418d71-plugins-config-volume" pod "bacfb744-b683-4b4d-a4e8-eefeaa418d71" (UID: "bacfb744-b683-4b4d-a4e8-eefeaa418d71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:15.320733 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.320697    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e20d7a0-a20d-48fb-a666-c5c89ec7a927-plugins-config-volume" pod "0e20d7a0-a20d-48fb-a666-c5c89ec7a927" (UID: "0e20d7a0-a20d-48fb-a666-c5c89ec7a927"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:15.327670 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.327621    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e20d7a0-a20d-48fb-a666-c5c89ec7a927-kube-api-access-v5pqc" pod "0e20d7a0-a20d-48fb-a666-c5c89ec7a927" (UID: "0e20d7a0-a20d-48fb-a666-c5c89ec7a927"). InnerVolumeSpecName "kube-api-access-v5pqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:15.329397 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.329357    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bacfb744-b683-4b4d-a4e8-eefeaa418d71-kube-api-access-wn9np" pod "bacfb744-b683-4b4d-a4e8-eefeaa418d71" (UID: "bacfb744-b683-4b4d-a4e8-eefeaa418d71"). InnerVolumeSpecName "kube-api-access-wn9np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:15.398955 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.397226    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:15.418104 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.417809    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn9np\" (UniqueName: \"kubernetes.io/projected/bacfb744-b683-4b4d-a4e8-eefeaa418d71-kube-api-access-wn9np\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:15.418104 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.417840    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bacfb744-b683-4b4d-a4e8-eefeaa418d71-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:15.418104 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.417852    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e20d7a0-a20d-48fb-a666-c5c89ec7a927-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:15.418104 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.417861    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5pqc\" (UniqueName: \"kubernetes.io/projected/0e20d7a0-a20d-48fb-a666-c5c89ec7a927-kube-api-access-v5pqc\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:15.457335 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.457300    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 14:00:15.468526 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.468438    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66"
Oct 31 14:00:15.471342 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.471296    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66" event={"ID":"d7298101-5b09-4055-964b-1457aeaa8c0f","Type":"ContainerDied","Data":"30a9ddf2d35fb2fd4cca65b650779f886a5ef4094b3e5914d8595e4d39b784f9"}
Oct 31 14:00:15.471487 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.471372    4428 scope.go:122] "RemoveContainer" containerID="dd475ac556e44bc16b0cd95594dc3c43d073a9e5f2f7d9cfb6fdc243f79c3647"
Oct 31 14:00:15.472477 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.472358    4428 generic.go:356] "Generic (PLEG): container finished" podID="b49565e4-99d7-4a99-a912-047c19c4b8b6" containerID="f43ef616cbca816e300225cc893aaaa0a295e275485e64df6500d71958668ea9" exitCode=60
Oct 31 14:00:15.472563 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.472504    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82" event={"ID":"b49565e4-99d7-4a99-a912-047c19c4b8b6","Type":"ContainerDied","Data":"f43ef616cbca816e300225cc893aaaa0a295e275485e64df6500d71958668ea9"}
Oct 31 14:00:15.473082 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.473053    4428 scope.go:122] "RemoveContainer" containerID="f43ef616cbca816e300225cc893aaaa0a295e275485e64df6500d71958668ea9"
Oct 31 14:00:15.473512 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:15.473481    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82_odh-ai-gateway-infra(b49565e4-99d7-4a99-a912-047c19c4b8b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82" podUID="b49565e4-99d7-4a99-a912-047c19c4b8b6"
Oct 31 14:00:15.474531 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.474504    4428 generic.go:356] "Generic (PLEG): container finished" podID="bacfb744-b683-4b4d-a4e8-eefeaa418d71" containerID="89e4713b3399c8f1fb2ab5be45a9db875aa234198c78679dbed3a2d9a1ee2200" exitCode=0
Oct 31 14:00:15.474652 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.474566    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"
Oct 31 14:00:15.474727 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.474677    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt" event={"ID":"bacfb744-b683-4b4d-a4e8-eefeaa418d71","Type":"ContainerDied","Data":"89e4713b3399c8f1fb2ab5be45a9db875aa234198c78679dbed3a2d9a1ee2200"}
Oct 31 14:00:15.474787 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.474725    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt" event={"ID":"bacfb744-b683-4b4d-a4e8-eefeaa418d71","Type":"ContainerDied","Data":"17056fce3292991dfb996e47e34ec9696ded1e75ebf13e915c3558e87259fb4c"}
Oct 31 14:00:15.476446 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.476422    4428 generic.go:356] "Generic (PLEG): container finished" podID="5c65ac2b-9ff4-4752-8325-bb8bd027708f" containerID="e1fa52666d1ee9f296e94708571020058a39196843ac1e221bd1b594946fbd9f" exitCode=0
Oct 31 14:00:15.476917 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.476645    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"
Oct 31 14:00:15.476917 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.476793    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq" event={"ID":"5c65ac2b-9ff4-4752-8325-bb8bd027708f","Type":"ContainerDied","Data":"e1fa52666d1ee9f296e94708571020058a39196843ac1e221bd1b594946fbd9f"}
Oct 31 14:00:15.476917 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.476831    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq" event={"ID":"5c65ac2b-9ff4-4752-8325-bb8bd027708f","Type":"ContainerDied","Data":"32dad84755df6206bb1c72abd79549a9599d85d4bd0cf13cc225da78cac1e70b"}
Oct 31 14:00:15.479849 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.479823    4428 generic.go:356] "Generic (PLEG): container finished" podID="0e20d7a0-a20d-48fb-a666-c5c89ec7a927" containerID="1b980d1002f58f4d44cb0ad875b7db658607cf113a782d703712ced96ca8e352" exitCode=0
Oct 31 14:00:15.480046 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.479915    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"
Oct 31 14:00:15.480637 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.479937    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz" event={"ID":"0e20d7a0-a20d-48fb-a666-c5c89ec7a927","Type":"ContainerDied","Data":"1b980d1002f58f4d44cb0ad875b7db658607cf113a782d703712ced96ca8e352"}
Oct 31 14:00:15.480637 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.480379    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz" event={"ID":"0e20d7a0-a20d-48fb-a666-c5c89ec7a927","Type":"ContainerDied","Data":"cf5e8042692d90e6cee6caab9ebafbcee19a9d4eb6705a5a3dc5ee1a00e2b852"}
Oct 31 14:00:15.496328 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.496250    4428 scope.go:122] "RemoveContainer" containerID="c3add6cf8867c509a378611c3373777e59ebcabed697631f38978b1e87cf293f"
Oct 31 14:00:15.502130 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.502085    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82"]
Oct 31 14:00:15.521725 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.521661    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c65ac2b-9ff4-4752-8325-bb8bd027708f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c65ac2b-9ff4-4752-8325-bb8bd027708f-metrics-certs\") pod \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\" (UID: \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\") "
Oct 31 14:00:15.522013 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.521848    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c65ac2b-9ff4-4752-8325-bb8bd027708f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c65ac2b-9ff4-4752-8325-bb8bd027708f-openshift-service-ca\") pod \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\" (UID: \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\") "
Oct 31 14:00:15.522013 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.521954    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c65ac2b-9ff4-4752-8325-bb8bd027708f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c65ac2b-9ff4-4752-8325-bb8bd027708f-maas-api-tls\") pod \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\" (UID: \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\") "
Oct 31 14:00:15.522250 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.522108    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c65ac2b-9ff4-4752-8325-bb8bd027708f-kube-api-access-z4wvl\" (UniqueName: \"kubernetes.io/projected/5c65ac2b-9ff4-4752-8325-bb8bd027708f-kube-api-access-z4wvl\") pod \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\" (UID: \"5c65ac2b-9ff4-4752-8325-bb8bd027708f\") "
Oct 31 14:00:15.529228 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.529203    4428 scope.go:122] "RemoveContainer" containerID="89e4713b3399c8f1fb2ab5be45a9db875aa234198c78679dbed3a2d9a1ee2200"
Oct 31 14:00:15.530654 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.530427    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c65ac2b-9ff4-4752-8325-bb8bd027708f-openshift-service-ca" pod "5c65ac2b-9ff4-4752-8325-bb8bd027708f" (UID: "5c65ac2b-9ff4-4752-8325-bb8bd027708f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:15.551428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.550422    4428 scope.go:122] "RemoveContainer" containerID="89e4713b3399c8f1fb2ab5be45a9db875aa234198c78679dbed3a2d9a1ee2200"
Oct 31 14:00:15.553939 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:15.553693    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e4713b3399c8f1fb2ab5be45a9db875aa234198c78679dbed3a2d9a1ee2200\": container with ID starting with 89e4713b3399c8f1fb2ab5be45a9db875aa234198c78679dbed3a2d9a1ee2200 not found: ID does not exist" containerID="89e4713b3399c8f1fb2ab5be45a9db875aa234198c78679dbed3a2d9a1ee2200"
Oct 31 14:00:15.553939 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.553738    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e4713b3399c8f1fb2ab5be45a9db875aa234198c78679dbed3a2d9a1ee2200"} err="failed to get container status \"89e4713b3399c8f1fb2ab5be45a9db875aa234198c78679dbed3a2d9a1ee2200\": rpc error: code = NotFound desc = could not find container \"89e4713b3399c8f1fb2ab5be45a9db875aa234198c78679dbed3a2d9a1ee2200\": container with ID starting with 89e4713b3399c8f1fb2ab5be45a9db875aa234198c78679dbed3a2d9a1ee2200 not found: ID does not exist"
Oct 31 14:00:15.553939 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.553767    4428 scope.go:122] "RemoveContainer" containerID="e1fa52666d1ee9f296e94708571020058a39196843ac1e221bd1b594946fbd9f"
Oct 31 14:00:15.564454 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.564033    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c65ac2b-9ff4-4752-8325-bb8bd027708f-metrics-certs" pod "5c65ac2b-9ff4-4752-8325-bb8bd027708f" (UID: "5c65ac2b-9ff4-4752-8325-bb8bd027708f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:00:15.565929 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.565860    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c65ac2b-9ff4-4752-8325-bb8bd027708f-kube-api-access-z4wvl" pod "5c65ac2b-9ff4-4752-8325-bb8bd027708f" (UID: "5c65ac2b-9ff4-4752-8325-bb8bd027708f"). InnerVolumeSpecName "kube-api-access-z4wvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:15.567005 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.566977    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c65ac2b-9ff4-4752-8325-bb8bd027708f-maas-api-tls" pod "5c65ac2b-9ff4-4752-8325-bb8bd027708f" (UID: "5c65ac2b-9ff4-4752-8325-bb8bd027708f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:00:15.593375 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.593316    4428 scope.go:122] "RemoveContainer" containerID="e1fa52666d1ee9f296e94708571020058a39196843ac1e221bd1b594946fbd9f"
Oct 31 14:00:15.594740 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:15.594550    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1fa52666d1ee9f296e94708571020058a39196843ac1e221bd1b594946fbd9f\": container with ID starting with e1fa52666d1ee9f296e94708571020058a39196843ac1e221bd1b594946fbd9f not found: ID does not exist" containerID="e1fa52666d1ee9f296e94708571020058a39196843ac1e221bd1b594946fbd9f"
Oct 31 14:00:15.594740 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.594597    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1fa52666d1ee9f296e94708571020058a39196843ac1e221bd1b594946fbd9f"} err="failed to get container status \"e1fa52666d1ee9f296e94708571020058a39196843ac1e221bd1b594946fbd9f\": rpc error: code = NotFound desc = could not find container \"e1fa52666d1ee9f296e94708571020058a39196843ac1e221bd1b594946fbd9f\": container with ID starting with e1fa52666d1ee9f296e94708571020058a39196843ac1e221bd1b594946fbd9f not found: ID does not exist"
Oct 31 14:00:15.594740 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.594628    4428 scope.go:122] "RemoveContainer" containerID="1b980d1002f58f4d44cb0ad875b7db658607cf113a782d703712ced96ca8e352"
Oct 31 14:00:15.594740 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.594636    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:15.596592 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.596416    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66"]
Oct 31 14:00:15.603265 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.603228    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5af6eb-29890920-kff66"]
Oct 31 14:00:15.612664 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.611423    4428 scope.go:122] "RemoveContainer" containerID="1b980d1002f58f4d44cb0ad875b7db658607cf113a782d703712ced96ca8e352"
Oct 31 14:00:15.613038 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:15.612935    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b980d1002f58f4d44cb0ad875b7db658607cf113a782d703712ced96ca8e352\": container with ID starting with 1b980d1002f58f4d44cb0ad875b7db658607cf113a782d703712ced96ca8e352 not found: ID does not exist" containerID="1b980d1002f58f4d44cb0ad875b7db658607cf113a782d703712ced96ca8e352"
Oct 31 14:00:15.613038 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.612996    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b980d1002f58f4d44cb0ad875b7db658607cf113a782d703712ced96ca8e352"} err="failed to get container status \"1b980d1002f58f4d44cb0ad875b7db658607cf113a782d703712ced96ca8e352\": rpc error: code = NotFound desc = could not find container \"1b980d1002f58f4d44cb0ad875b7db658607cf113a782d703712ced96ca8e352\": container with ID starting with 1b980d1002f58f4d44cb0ad875b7db658607cf113a782d703712ced96ca8e352 not found: ID does not exist"
Oct 31 14:00:15.671309 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.671265    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4wvl\" (UniqueName: \"kubernetes.io/projected/5c65ac2b-9ff4-4752-8325-bb8bd027708f-kube-api-access-z4wvl\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:15.671587 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.671567    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c65ac2b-9ff4-4752-8325-bb8bd027708f-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:15.671711 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.671681    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c65ac2b-9ff4-4752-8325-bb8bd027708f-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:15.671812 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.671799    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c65ac2b-9ff4-4752-8325-bb8bd027708f-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:15.688755 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.688706    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"]
Oct 31 14:00:15.695606 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.695567    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5af6eb-8459557758-bl7fz"]
Oct 31 14:00:15.706130 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.706090    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"]
Oct 31 14:00:15.712047 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.712018    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5af6eb-76b9475667-xphdt"]
Oct 31 14:00:15.748307 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.748214    4428 scope.go:122] "RemoveContainer" containerID="5dd3c77c1bab081af45ef63a477e39bab88db81a753b0fa007f731d33b5d554d"
Oct 31 14:00:15.748499 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.748462    4428 scope.go:122] "RemoveContainer" containerID="10e6e30801163e636c73dd52ff6dc04dd9b55024294bc3e692f80e5bc5ba8995"
Oct 31 14:00:15.784118 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.783527    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:15.810610 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.810560    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"]
Oct 31 14:00:15.817192 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.817155    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a81979-d5f74d6dc-rlqkq"]
Oct 31 14:00:15.984202 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:15.984168    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:16.292223 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:16.292185    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:16.519536 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:16.519498    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q" event={"ID":"9ab8637d-ccad-4855-b7ff-455c744f3da5","Type":"ContainerStarted","Data":"4d4fcf65350e354e6dd015d9994b7bd8b74a76a4d7e6c2f507d5aa7e3fd8fc93"}
Oct 31 14:00:16.523980 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:16.523947    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg" event={"ID":"124e886b-210d-465f-b6a2-5dcd03b03457","Type":"ContainerStarted","Data":"338de078a1fbe587d18611fd75cd495d1b98ba363c71cce0164fa2065696313e"}
Oct 31 14:00:16.675892 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:16.667760    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:16.748956 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:16.748923    4428 scope.go:122] "RemoveContainer" containerID="e5cc63e1d4833c44ad12ef2600718628d6d87df353e47b9163674b0e99cdae8a"
Oct 31 14:00:16.749237 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:16.749203    4428 scope.go:122] "RemoveContainer" containerID="a8a160ee900877575e32f4c337e2090822afefb6bb76443f8660ad9d114578a3"
Oct 31 14:00:16.749389 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:16.749364    4428 scope.go:122] "RemoveContainer" containerID="e4a57f40a97c1350a0f761a616d54a95492dc170c98db044d9679231188019d4"
Oct 31 14:00:16.765022 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:16.764995    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e20d7a0-a20d-48fb-a666-c5c89ec7a927" path="/var/lib/kubelet/pods/0e20d7a0-a20d-48fb-a666-c5c89ec7a927/volumes"
Oct 31 14:00:16.766335 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:16.765658    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c65ac2b-9ff4-4752-8325-bb8bd027708f" path="/var/lib/kubelet/pods/5c65ac2b-9ff4-4752-8325-bb8bd027708f/volumes"
Oct 31 14:00:16.767662 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:16.767051    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bacfb744-b683-4b4d-a4e8-eefeaa418d71" path="/var/lib/kubelet/pods/bacfb744-b683-4b4d-a4e8-eefeaa418d71/volumes"
Oct 31 14:00:16.768700 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:16.768632    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7298101-5b09-4055-964b-1457aeaa8c0f" path="/var/lib/kubelet/pods/d7298101-5b09-4055-964b-1457aeaa8c0f/volumes"
Oct 31 14:00:16.943813 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:16.943777    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82"
Oct 31 14:00:16.995434 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:16.995400    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49565e4-99d7-4a99-a912-047c19c4b8b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49565e4-99d7-4a99-a912-047c19c4b8b6-maas-api-service-ca\") pod \"b49565e4-99d7-4a99-a912-047c19c4b8b6\" (UID: \"b49565e4-99d7-4a99-a912-047c19c4b8b6\") "
Oct 31 14:00:16.995663 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:16.995539    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49565e4-99d7-4a99-a912-047c19c4b8b6-kube-api-access-zxlrx\" (UniqueName: \"kubernetes.io/projected/b49565e4-99d7-4a99-a912-047c19c4b8b6-kube-api-access-zxlrx\") pod \"b49565e4-99d7-4a99-a912-047c19c4b8b6\" (UID: \"b49565e4-99d7-4a99-a912-047c19c4b8b6\") "
Oct 31 14:00:16.995935 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:16.995905    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49565e4-99d7-4a99-a912-047c19c4b8b6-maas-api-service-ca" pod "b49565e4-99d7-4a99-a912-047c19c4b8b6" (UID: "b49565e4-99d7-4a99-a912-047c19c4b8b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:16.996229 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:16.996206    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49565e4-99d7-4a99-a912-047c19c4b8b6-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:16.998160 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:16.998123    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49565e4-99d7-4a99-a912-047c19c4b8b6-kube-api-access-zxlrx" pod "b49565e4-99d7-4a99-a912-047c19c4b8b6" (UID: "b49565e4-99d7-4a99-a912-047c19c4b8b6"). InnerVolumeSpecName "kube-api-access-zxlrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:17.097740 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.097648    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxlrx\" (UniqueName: \"kubernetes.io/projected/b49565e4-99d7-4a99-a912-047c19c4b8b6-kube-api-access-zxlrx\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:17.541444 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.541408    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp" event={"ID":"05b7941b-bb75-4e52-8766-7f7eb2f96734","Type":"ContainerStarted","Data":"b56a864e4d9dd63ba173bb8bbc890a8676345afd174c076e568b0a85b6b7c48a"}
Oct 31 14:00:17.543331 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.543304    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82"
Oct 31 14:00:17.543487 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.543340    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82" event={"ID":"b49565e4-99d7-4a99-a912-047c19c4b8b6","Type":"ContainerDied","Data":"b2e9b589ba6d450faf0c5fed7adab81729db687f6a802aff02be7e39b52476b0"}
Oct 31 14:00:17.543487 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.543406    4428 scope.go:122] "RemoveContainer" containerID="f43ef616cbca816e300225cc893aaaa0a295e275485e64df6500d71958668ea9"
Oct 31 14:00:17.548052 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.547993    4428 generic.go:356] "Generic (PLEG): container finished" podID="9ab8637d-ccad-4855-b7ff-455c744f3da5" containerID="4d4fcf65350e354e6dd015d9994b7bd8b74a76a4d7e6c2f507d5aa7e3fd8fc93" exitCode=60
Oct 31 14:00:17.548232 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.548129    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q" event={"ID":"9ab8637d-ccad-4855-b7ff-455c744f3da5","Type":"ContainerDied","Data":"4d4fcf65350e354e6dd015d9994b7bd8b74a76a4d7e6c2f507d5aa7e3fd8fc93"}
Oct 31 14:00:17.548617 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.548593    4428 scope.go:122] "RemoveContainer" containerID="4d4fcf65350e354e6dd015d9994b7bd8b74a76a4d7e6c2f507d5aa7e3fd8fc93"
Oct 31 14:00:17.549327 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:17.549299    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q_odh-ai-gateway-infra(9ab8637d-ccad-4855-b7ff-455c744f3da5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q" podUID="9ab8637d-ccad-4855-b7ff-455c744f3da5"
Oct 31 14:00:17.555422 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.555388    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n" event={"ID":"0fee8a48-bc4e-4df9-b4db-354d2f577fd0","Type":"ContainerStarted","Data":"f9e551c73213e48af7ab4d3ddc1712ef5dc1f91834ce1bf99616c9da1cd0ff72"}
Oct 31 14:00:17.558047 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.558018    4428 generic.go:356] "Generic (PLEG): container finished" podID="124e886b-210d-465f-b6a2-5dcd03b03457" containerID="338de078a1fbe587d18611fd75cd495d1b98ba363c71cce0164fa2065696313e" exitCode=60
Oct 31 14:00:17.558256 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.558168    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg" event={"ID":"124e886b-210d-465f-b6a2-5dcd03b03457","Type":"ContainerDied","Data":"338de078a1fbe587d18611fd75cd495d1b98ba363c71cce0164fa2065696313e"}
Oct 31 14:00:17.558606 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.558587    4428 scope.go:122] "RemoveContainer" containerID="338de078a1fbe587d18611fd75cd495d1b98ba363c71cce0164fa2065696313e"
Oct 31 14:00:17.558720 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.558698    4428 scope.go:122] "RemoveContainer" containerID="5dd3c77c1bab081af45ef63a477e39bab88db81a753b0fa007f731d33b5d554d"
Oct 31 14:00:17.558971 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:17.558952    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg_odh-ai-gateway-infra(124e886b-210d-465f-b6a2-5dcd03b03457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg" podUID="124e886b-210d-465f-b6a2-5dcd03b03457"
Oct 31 14:00:17.578058 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.578022    4428 scope.go:122] "RemoveContainer" containerID="10e6e30801163e636c73dd52ff6dc04dd9b55024294bc3e692f80e5bc5ba8995"
Oct 31 14:00:17.579267 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.579236    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg"]
Oct 31 14:00:17.592105 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.592079    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82"]
Oct 31 14:00:17.599215 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.599191    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q"]
Oct 31 14:00:17.641724 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.641439    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:17.661447 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:17.660924    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97927b-29890920-m2n82"]
Oct 31 14:00:18.524269 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.524052    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:18.571452 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.571421    4428 generic.go:356] "Generic (PLEG): container finished" podID="c2da70ae-abad-42d5-be21-0fd2b83848e0" containerID="36e54d51c0030b7bffd4c8fe58c4d90e35019d908827fbbca0b30fa559097371" exitCode=0
Oct 31 14:00:18.571941 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.571491    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt" event={"ID":"c2da70ae-abad-42d5-be21-0fd2b83848e0","Type":"ContainerDied","Data":"36e54d51c0030b7bffd4c8fe58c4d90e35019d908827fbbca0b30fa559097371"}
Oct 31 14:00:18.572132 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.572112    4428 scope.go:122] "RemoveContainer" containerID="36e54d51c0030b7bffd4c8fe58c4d90e35019d908827fbbca0b30fa559097371"
Oct 31 14:00:18.574379 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.574320    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:00:18.574461 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.574414    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:00:18.574526 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.574468    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:00:18.576857 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.576826    4428 generic.go:356] "Generic (PLEG): container finished" podID="0fee8a48-bc4e-4df9-b4db-354d2f577fd0" containerID="f9e551c73213e48af7ab4d3ddc1712ef5dc1f91834ce1bf99616c9da1cd0ff72" exitCode=60
Oct 31 14:00:18.577005 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.576940    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n" event={"ID":"0fee8a48-bc4e-4df9-b4db-354d2f577fd0","Type":"ContainerDied","Data":"f9e551c73213e48af7ab4d3ddc1712ef5dc1f91834ce1bf99616c9da1cd0ff72"}
Oct 31 14:00:18.577070 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.577022    4428 scope.go:122] "RemoveContainer" containerID="e4a57f40a97c1350a0f761a616d54a95492dc170c98db044d9679231188019d4"
Oct 31 14:00:18.577624 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.577604    4428 scope.go:122] "RemoveContainer" containerID="f9e551c73213e48af7ab4d3ddc1712ef5dc1f91834ce1bf99616c9da1cd0ff72"
Oct 31 14:00:18.578540 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:18.578507    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n_odh-ai-gateway-infra(0fee8a48-bc4e-4df9-b4db-354d2f577fd0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n" podUID="0fee8a48-bc4e-4df9-b4db-354d2f577fd0"
Oct 31 14:00:18.582184 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.582157    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp"]
Oct 31 14:00:18.584750 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.584715    4428 generic.go:356] "Generic (PLEG): container finished" podID="f0e18350-e8e5-4d56-89e9-dc8dbdf12de8" containerID="0b2c9d336ce0bc3318aee53526cf9bf9c9d664c841d0d2bbd7f4c28533589e69" exitCode=60
Oct 31 14:00:18.584964 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.584939    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp" event={"ID":"f0e18350-e8e5-4d56-89e9-dc8dbdf12de8","Type":"ContainerDied","Data":"0b2c9d336ce0bc3318aee53526cf9bf9c9d664c841d0d2bbd7f4c28533589e69"}
Oct 31 14:00:18.585546 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.585528    4428 scope.go:122] "RemoveContainer" containerID="0b2c9d336ce0bc3318aee53526cf9bf9c9d664c841d0d2bbd7f4c28533589e69"
Oct 31 14:00:18.585978 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:18.585954    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp_odh-ai-gateway-infra(f0e18350-e8e5-4d56-89e9-dc8dbdf12de8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp" podUID="f0e18350-e8e5-4d56-89e9-dc8dbdf12de8"
Oct 31 14:00:18.589216 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.589191    4428 generic.go:356] "Generic (PLEG): container finished" podID="05b7941b-bb75-4e52-8766-7f7eb2f96734" containerID="b56a864e4d9dd63ba173bb8bbc890a8676345afd174c076e568b0a85b6b7c48a" exitCode=60
Oct 31 14:00:18.589314 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.589284    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp" event={"ID":"05b7941b-bb75-4e52-8766-7f7eb2f96734","Type":"ContainerDied","Data":"b56a864e4d9dd63ba173bb8bbc890a8676345afd174c076e568b0a85b6b7c48a"}
Oct 31 14:00:18.613513 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.613481    4428 scope.go:122] "RemoveContainer" containerID="e5cc63e1d4833c44ad12ef2600718628d6d87df353e47b9163674b0e99cdae8a"
Oct 31 14:00:18.682182 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.680729    4428 scope.go:122] "RemoveContainer" containerID="a8a160ee900877575e32f4c337e2090822afefb6bb76443f8660ad9d114578a3"
Oct 31 14:00:18.691035 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.690924    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n"]
Oct 31 14:00:18.787567 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.787525    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49565e4-99d7-4a99-a912-047c19c4b8b6" path="/var/lib/kubelet/pods/b49565e4-99d7-4a99-a912-047c19c4b8b6/volumes"
Oct 31 14:00:18.795919 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:18.795591    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:00:18.860800 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.860714    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q"
Oct 31 14:00:18.933968 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.933942    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg"
Oct 31 14:00:18.944910 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.938982    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab8637d-ccad-4855-b7ff-455c744f3da5-kube-api-access-5sqhz\" (UniqueName: \"kubernetes.io/projected/9ab8637d-ccad-4855-b7ff-455c744f3da5-kube-api-access-5sqhz\") pod \"9ab8637d-ccad-4855-b7ff-455c744f3da5\" (UID: \"9ab8637d-ccad-4855-b7ff-455c744f3da5\") "
Oct 31 14:00:18.944910 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.939153    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ab8637d-ccad-4855-b7ff-455c744f3da5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab8637d-ccad-4855-b7ff-455c744f3da5-maas-api-service-ca\") pod \"9ab8637d-ccad-4855-b7ff-455c744f3da5\" (UID: \"9ab8637d-ccad-4855-b7ff-455c744f3da5\") "
Oct 31 14:00:18.944910 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.939955    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab8637d-ccad-4855-b7ff-455c744f3da5-maas-api-service-ca" pod "9ab8637d-ccad-4855-b7ff-455c744f3da5" (UID: "9ab8637d-ccad-4855-b7ff-455c744f3da5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:18.944910 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.941573    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab8637d-ccad-4855-b7ff-455c744f3da5-kube-api-access-5sqhz" pod "9ab8637d-ccad-4855-b7ff-455c744f3da5" (UID: "9ab8637d-ccad-4855-b7ff-455c744f3da5"). InnerVolumeSpecName "kube-api-access-5sqhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:18.944910 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:18.943186    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp"
Oct 31 14:00:19.041742 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.041701    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124e886b-210d-465f-b6a2-5dcd03b03457-kube-api-access-2hcfc\" (UniqueName: \"kubernetes.io/projected/124e886b-210d-465f-b6a2-5dcd03b03457-kube-api-access-2hcfc\") pod \"124e886b-210d-465f-b6a2-5dcd03b03457\" (UID: \"124e886b-210d-465f-b6a2-5dcd03b03457\") "
Oct 31 14:00:19.041742 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.041753    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05b7941b-bb75-4e52-8766-7f7eb2f96734-kube-api-access-w8wp5\" (UniqueName: \"kubernetes.io/projected/05b7941b-bb75-4e52-8766-7f7eb2f96734-kube-api-access-w8wp5\") pod \"05b7941b-bb75-4e52-8766-7f7eb2f96734\" (UID: \"05b7941b-bb75-4e52-8766-7f7eb2f96734\") "
Oct 31 14:00:19.042098 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.041837    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/124e886b-210d-465f-b6a2-5dcd03b03457-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/124e886b-210d-465f-b6a2-5dcd03b03457-maas-api-service-ca\") pod \"124e886b-210d-465f-b6a2-5dcd03b03457\" (UID: \"124e886b-210d-465f-b6a2-5dcd03b03457\") "
Oct 31 14:00:19.042098 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.041918    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05b7941b-bb75-4e52-8766-7f7eb2f96734-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05b7941b-bb75-4e52-8766-7f7eb2f96734-maas-api-service-ca\") pod \"05b7941b-bb75-4e52-8766-7f7eb2f96734\" (UID: \"05b7941b-bb75-4e52-8766-7f7eb2f96734\") "
Oct 31 14:00:19.042890 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.042847    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sqhz\" (UniqueName: \"kubernetes.io/projected/9ab8637d-ccad-4855-b7ff-455c744f3da5-kube-api-access-5sqhz\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:19.043469 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.043450    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab8637d-ccad-4855-b7ff-455c744f3da5-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:19.043611 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.043404    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05b7941b-bb75-4e52-8766-7f7eb2f96734-maas-api-service-ca" pod "05b7941b-bb75-4e52-8766-7f7eb2f96734" (UID: "05b7941b-bb75-4e52-8766-7f7eb2f96734"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:19.044920 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.044893    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/124e886b-210d-465f-b6a2-5dcd03b03457-maas-api-service-ca" pod "124e886b-210d-465f-b6a2-5dcd03b03457" (UID: "124e886b-210d-465f-b6a2-5dcd03b03457"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:19.045392 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.045371    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b7941b-bb75-4e52-8766-7f7eb2f96734-kube-api-access-w8wp5" pod "05b7941b-bb75-4e52-8766-7f7eb2f96734" (UID: "05b7941b-bb75-4e52-8766-7f7eb2f96734"). InnerVolumeSpecName "kube-api-access-w8wp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:19.045481 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.045448    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124e886b-210d-465f-b6a2-5dcd03b03457-kube-api-access-2hcfc" pod "124e886b-210d-465f-b6a2-5dcd03b03457" (UID: "124e886b-210d-465f-b6a2-5dcd03b03457"). InnerVolumeSpecName "kube-api-access-2hcfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:19.145375 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.145286    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hcfc\" (UniqueName: \"kubernetes.io/projected/124e886b-210d-465f-b6a2-5dcd03b03457-kube-api-access-2hcfc\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:19.145375 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.145316    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8wp5\" (UniqueName: \"kubernetes.io/projected/05b7941b-bb75-4e52-8766-7f7eb2f96734-kube-api-access-w8wp5\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:19.145375 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.145341    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/124e886b-210d-465f-b6a2-5dcd03b03457-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:19.145375 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.145350    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05b7941b-bb75-4e52-8766-7f7eb2f96734-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:19.466127 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.466088    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:19.600534 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.600494    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt" event={"ID":"c2da70ae-abad-42d5-be21-0fd2b83848e0","Type":"ContainerStarted","Data":"9dbd592ed3119777464e88f2bad4f0d069d587b80fde38640207c8c951a7cc93"}
Oct 31 14:00:19.602230 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.602194    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q"
Oct 31 14:00:19.602406 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.602245    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q" event={"ID":"9ab8637d-ccad-4855-b7ff-455c744f3da5","Type":"ContainerDied","Data":"19fe0381e3525c22168d52d9c2afb637456568bd58428b40ddd4ba957594b4ab"}
Oct 31 14:00:19.602406 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.602316    4428 scope.go:122] "RemoveContainer" containerID="4d4fcf65350e354e6dd015d9994b7bd8b74a76a4d7e6c2f507d5aa7e3fd8fc93"
Oct 31 14:00:19.608446 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.608413    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg"
Oct 31 14:00:19.608576 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.608437    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg" event={"ID":"124e886b-210d-465f-b6a2-5dcd03b03457","Type":"ContainerDied","Data":"afdf21f13f13af53f104d0a363edf24e6b44f968b86bf85890843f6cdc49e0ca"}
Oct 31 14:00:19.610132 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.610109    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp" event={"ID":"05b7941b-bb75-4e52-8766-7f7eb2f96734","Type":"ContainerDied","Data":"6b7ec85ce3418c84c941dda438278f7f5382f6c88fd7533f502fb2ef0538913f"}
Oct 31 14:00:19.610251 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.610165    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp"
Oct 31 14:00:19.616688 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.616650    4428 scope.go:122] "RemoveContainer" containerID="338de078a1fbe587d18611fd75cd495d1b98ba363c71cce0164fa2065696313e"
Oct 31 14:00:19.633489 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.633456    4428 scope.go:122] "RemoveContainer" containerID="b56a864e4d9dd63ba173bb8bbc890a8676345afd174c076e568b0a85b6b7c48a"
Oct 31 14:00:19.695396 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.695356    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg"]
Oct 31 14:00:19.704053 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.704002    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp"]
Oct 31 14:00:19.711748 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.711476    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abdcb1-29890920-bkdtg"]
Oct 31 14:00:19.724783 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.724662    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp"]
Oct 31 14:00:19.739302 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.738893    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890920-7r5gp"]
Oct 31 14:00:19.786517 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.786117    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q"]
Oct 31 14:00:19.791262 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.791229    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6784d6-29890920-tvz2q"]
Oct 31 14:00:19.902282 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:19.902251    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n"
Oct 31 14:00:20.007961 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.003696    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp"
Oct 31 14:00:20.094497 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.094460    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fee8a48-bc4e-4df9-b4db-354d2f577fd0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fee8a48-bc4e-4df9-b4db-354d2f577fd0-maas-api-service-ca\") pod \"0fee8a48-bc4e-4df9-b4db-354d2f577fd0\" (UID: \"0fee8a48-bc4e-4df9-b4db-354d2f577fd0\") "
Oct 31 14:00:20.094720 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.094517    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fee8a48-bc4e-4df9-b4db-354d2f577fd0-kube-api-access-f7rjf\" (UniqueName: \"kubernetes.io/projected/0fee8a48-bc4e-4df9-b4db-354d2f577fd0-kube-api-access-f7rjf\") pod \"0fee8a48-bc4e-4df9-b4db-354d2f577fd0\" (UID: \"0fee8a48-bc4e-4df9-b4db-354d2f577fd0\") "
Oct 31 14:00:20.094835 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.094811    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fee8a48-bc4e-4df9-b4db-354d2f577fd0-maas-api-service-ca" pod "0fee8a48-bc4e-4df9-b4db-354d2f577fd0" (UID: "0fee8a48-bc4e-4df9-b4db-354d2f577fd0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:20.095370 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.095345    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fee8a48-bc4e-4df9-b4db-354d2f577fd0-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:20.096843 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.096815    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fee8a48-bc4e-4df9-b4db-354d2f577fd0-kube-api-access-f7rjf" pod "0fee8a48-bc4e-4df9-b4db-354d2f577fd0" (UID: "0fee8a48-bc4e-4df9-b4db-354d2f577fd0"). InnerVolumeSpecName "kube-api-access-f7rjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:20.195962 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.195922    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e18350-e8e5-4d56-89e9-dc8dbdf12de8-kube-api-access-tz8gt\" (UniqueName: \"kubernetes.io/projected/f0e18350-e8e5-4d56-89e9-dc8dbdf12de8-kube-api-access-tz8gt\") pod \"f0e18350-e8e5-4d56-89e9-dc8dbdf12de8\" (UID: \"f0e18350-e8e5-4d56-89e9-dc8dbdf12de8\") "
Oct 31 14:00:20.196251 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.196229    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0e18350-e8e5-4d56-89e9-dc8dbdf12de8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e18350-e8e5-4d56-89e9-dc8dbdf12de8-maas-api-service-ca\") pod \"f0e18350-e8e5-4d56-89e9-dc8dbdf12de8\" (UID: \"f0e18350-e8e5-4d56-89e9-dc8dbdf12de8\") "
Oct 31 14:00:20.196636 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.196582    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e18350-e8e5-4d56-89e9-dc8dbdf12de8-maas-api-service-ca" pod "f0e18350-e8e5-4d56-89e9-dc8dbdf12de8" (UID: "f0e18350-e8e5-4d56-89e9-dc8dbdf12de8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:20.196962 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.196930    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7rjf\" (UniqueName: \"kubernetes.io/projected/0fee8a48-bc4e-4df9-b4db-354d2f577fd0-kube-api-access-f7rjf\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:20.196962 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.196947    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e18350-e8e5-4d56-89e9-dc8dbdf12de8-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:20.198525 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.198487    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e18350-e8e5-4d56-89e9-dc8dbdf12de8-kube-api-access-tz8gt" pod "f0e18350-e8e5-4d56-89e9-dc8dbdf12de8" (UID: "f0e18350-e8e5-4d56-89e9-dc8dbdf12de8"). InnerVolumeSpecName "kube-api-access-tz8gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:20.302953 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.301269    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz8gt\" (UniqueName: \"kubernetes.io/projected/f0e18350-e8e5-4d56-89e9-dc8dbdf12de8-kube-api-access-tz8gt\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:20.377908 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.377799    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:20.398001 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.397972    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"
Oct 31 14:00:20.425716 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.425684    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 14:00:20.427182 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.427151    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 31 14:00:20.427352 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.427209    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt" podUID="c2da70ae-abad-42d5-be21-0fd2b83848e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 31 14:00:20.620177 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.620086    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n"
Oct 31 14:00:20.620642 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.620082    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n" event={"ID":"0fee8a48-bc4e-4df9-b4db-354d2f577fd0","Type":"ContainerDied","Data":"6556d7390c82a921ed905ce55fc1a44fae8c8f573f089a06bc650a08780a984a"}
Oct 31 14:00:20.620642 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.620255    4428 scope.go:122] "RemoveContainer" containerID="f9e551c73213e48af7ab4d3ddc1712ef5dc1f91834ce1bf99616c9da1cd0ff72"
Oct 31 14:00:20.621688 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.621663    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp" event={"ID":"f0e18350-e8e5-4d56-89e9-dc8dbdf12de8","Type":"ContainerDied","Data":"2b590f875cf37e2b5bb7cb4616626794726f83f2812a31dd1cd49fb596b814a7"}
Oct 31 14:00:20.621688 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.621679    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp"
Oct 31 14:00:20.665897 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.645935    4428 scope.go:122] "RemoveContainer" containerID="0b2c9d336ce0bc3318aee53526cf9bf9c9d664c841d0d2bbd7f4c28533589e69"
Oct 31 14:00:20.684560 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.684504    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp"]
Oct 31 14:00:20.692520 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.692490    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890920-5svmp"]
Oct 31 14:00:20.698153 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.698128    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n"]
Oct 31 14:00:20.707665 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.707637    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890920-h6v8n"]
Oct 31 14:00:20.779504 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.779461    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05b7941b-bb75-4e52-8766-7f7eb2f96734" path="/var/lib/kubelet/pods/05b7941b-bb75-4e52-8766-7f7eb2f96734/volumes"
Oct 31 14:00:20.780008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.779993    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fee8a48-bc4e-4df9-b4db-354d2f577fd0" path="/var/lib/kubelet/pods/0fee8a48-bc4e-4df9-b4db-354d2f577fd0/volumes"
Oct 31 14:00:20.780426 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.780413    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="124e886b-210d-465f-b6a2-5dcd03b03457" path="/var/lib/kubelet/pods/124e886b-210d-465f-b6a2-5dcd03b03457/volumes"
Oct 31 14:00:20.781149 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.781134    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ab8637d-ccad-4855-b7ff-455c744f3da5" path="/var/lib/kubelet/pods/9ab8637d-ccad-4855-b7ff-455c744f3da5/volumes"
Oct 31 14:00:20.781528 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.781515    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0e18350-e8e5-4d56-89e9-dc8dbdf12de8" path="/var/lib/kubelet/pods/f0e18350-e8e5-4d56-89e9-dc8dbdf12de8/volumes"
Oct 31 14:00:20.875031 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:20.874949    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"]
Oct 31 14:00:21.030621 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:21.030588    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:21.326995 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:21.326653    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:21.425846 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:21.425805    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 31 14:00:21.426057 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:21.425885    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt" podUID="c2da70ae-abad-42d5-be21-0fd2b83848e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 31 14:00:22.100836 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:22.100793    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:22.390984 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:22.390853    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 14:00:22.425723 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:22.425690    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 31 14:00:22.425936 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:22.425746    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt" podUID="c2da70ae-abad-42d5-be21-0fd2b83848e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 31 14:00:22.819699 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:22.819662    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:23.009373 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:23.009339    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:23.095984 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:23.095898    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"]
Oct 31 14:00:23.096486 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:23.096456    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj" podUID="6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" containerName="istio-proxy" containerID="cri-o://0a96c1ca088318c212b57afa1adb59adf5f2fd39b61612c5bbb7eab5d1d04d1d" gracePeriod=30
Oct 31 14:00:23.209902 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:23.202471    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:23.425584 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:23.425502    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 31 14:00:23.425584 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:23.425555    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt" podUID="c2da70ae-abad-42d5-be21-0fd2b83848e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 31 14:00:24.132397 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:24.132365    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:24.425419 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:24.425326    4428 patch_prober.go:29] interesting pod/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Oct 31 14:00:24.425419 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:24.425388    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt" podUID="c2da70ae-abad-42d5-be21-0fd2b83848e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Oct 31 14:00:25.246893 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.245403    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:25.430656 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.430619    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 14:00:25.432003 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.431977    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 14:00:25.604819 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.604738    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"]
Oct 31 14:00:25.605539 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605524    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b49565e4-99d7-4a99-a912-047c19c4b8b6" containerName="cleanup"
Oct 31 14:00:25.605590 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605542    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49565e4-99d7-4a99-a912-047c19c4b8b6" containerName="cleanup"
Oct 31 14:00:25.605590 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605555    4428 cpu_manager.go:414] "containerMap: removing container" podUID="05b7941b-bb75-4e52-8766-7f7eb2f96734" containerName="cleanup"
Oct 31 14:00:25.605590 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605559    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05b7941b-bb75-4e52-8766-7f7eb2f96734" containerName="cleanup"
Oct 31 14:00:25.605590 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605569    4428 cpu_manager.go:414] "containerMap: removing container" podUID="124e886b-210d-465f-b6a2-5dcd03b03457" containerName="cleanup"
Oct 31 14:00:25.605590 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605574    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124e886b-210d-465f-b6a2-5dcd03b03457" containerName="cleanup"
Oct 31 14:00:25.605590 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605586    4428 cpu_manager.go:414] "containerMap: removing container" podUID="f0e18350-e8e5-4d56-89e9-dc8dbdf12de8" containerName="cleanup"
Oct 31 14:00:25.605590 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605591    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e18350-e8e5-4d56-89e9-dc8dbdf12de8" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605601    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b49565e4-99d7-4a99-a912-047c19c4b8b6" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605608    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49565e4-99d7-4a99-a912-047c19c4b8b6" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605614    4428 cpu_manager.go:414] "containerMap: removing container" podUID="0fee8a48-bc4e-4df9-b4db-354d2f577fd0" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605618    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fee8a48-bc4e-4df9-b4db-354d2f577fd0" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605626    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9ab8637d-ccad-4855-b7ff-455c744f3da5" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605630    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab8637d-ccad-4855-b7ff-455c744f3da5" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605637    4428 cpu_manager.go:414] "containerMap: removing container" podUID="0e20d7a0-a20d-48fb-a666-c5c89ec7a927" containerName="payload-processing"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605641    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e20d7a0-a20d-48fb-a666-c5c89ec7a927" containerName="payload-processing"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605648    4428 cpu_manager.go:414] "containerMap: removing container" podUID="0fee8a48-bc4e-4df9-b4db-354d2f577fd0" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605653    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fee8a48-bc4e-4df9-b4db-354d2f577fd0" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605671    4428 cpu_manager.go:414] "containerMap: removing container" podUID="5c65ac2b-9ff4-4752-8325-bb8bd027708f" containerName="maas-api"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605676    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c65ac2b-9ff4-4752-8325-bb8bd027708f" containerName="maas-api"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605683    4428 cpu_manager.go:414] "containerMap: removing container" podUID="bacfb744-b683-4b4d-a4e8-eefeaa418d71" containerName="payload-pre-processing"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605687    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bacfb744-b683-4b4d-a4e8-eefeaa418d71" containerName="payload-pre-processing"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605693    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d7298101-5b09-4055-964b-1457aeaa8c0f" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605698    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7298101-5b09-4055-964b-1457aeaa8c0f" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605705    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d7298101-5b09-4055-964b-1457aeaa8c0f" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605709    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7298101-5b09-4055-964b-1457aeaa8c0f" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605717    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b49565e4-99d7-4a99-a912-047c19c4b8b6" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605722    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49565e4-99d7-4a99-a912-047c19c4b8b6" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605728    4428 cpu_manager.go:414] "containerMap: removing container" podUID="f0e18350-e8e5-4d56-89e9-dc8dbdf12de8" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605732    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e18350-e8e5-4d56-89e9-dc8dbdf12de8" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605740    4428 cpu_manager.go:414] "containerMap: removing container" podUID="124e886b-210d-465f-b6a2-5dcd03b03457" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605743    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124e886b-210d-465f-b6a2-5dcd03b03457" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605760    4428 cpu_manager.go:414] "containerMap: removing container" podUID="f0e18350-e8e5-4d56-89e9-dc8dbdf12de8" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605765    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e18350-e8e5-4d56-89e9-dc8dbdf12de8" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605774    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9ab8637d-ccad-4855-b7ff-455c744f3da5" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605778    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab8637d-ccad-4855-b7ff-455c744f3da5" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605785    4428 cpu_manager.go:414] "containerMap: removing container" podUID="0fee8a48-bc4e-4df9-b4db-354d2f577fd0" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605789    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fee8a48-bc4e-4df9-b4db-354d2f577fd0" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605798    4428 cpu_manager.go:414] "containerMap: removing container" podUID="124e886b-210d-465f-b6a2-5dcd03b03457" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605802    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124e886b-210d-465f-b6a2-5dcd03b03457" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605811    4428 cpu_manager.go:414] "containerMap: removing container" podUID="05b7941b-bb75-4e52-8766-7f7eb2f96734" containerName="cleanup"
Oct 31 14:00:25.605814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605815    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05b7941b-bb75-4e52-8766-7f7eb2f96734" containerName="cleanup"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605933    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="124e886b-210d-465f-b6a2-5dcd03b03457" containerName="cleanup"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605943    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e18350-e8e5-4d56-89e9-dc8dbdf12de8" containerName="cleanup"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605950    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49565e4-99d7-4a99-a912-047c19c4b8b6" containerName="cleanup"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605955    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fee8a48-bc4e-4df9-b4db-354d2f577fd0" containerName="cleanup"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605960    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab8637d-ccad-4855-b7ff-455c744f3da5" containerName="cleanup"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605966    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e20d7a0-a20d-48fb-a666-c5c89ec7a927" containerName="payload-processing"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605971    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fee8a48-bc4e-4df9-b4db-354d2f577fd0" containerName="cleanup"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605979    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7298101-5b09-4055-964b-1457aeaa8c0f" containerName="cleanup"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605986    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="bacfb744-b683-4b4d-a4e8-eefeaa418d71" containerName="payload-pre-processing"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605990    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c65ac2b-9ff4-4752-8325-bb8bd027708f" containerName="maas-api"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.605995    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7298101-5b09-4055-964b-1457aeaa8c0f" containerName="cleanup"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.606000    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab8637d-ccad-4855-b7ff-455c744f3da5" containerName="cleanup"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.606004    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e18350-e8e5-4d56-89e9-dc8dbdf12de8" containerName="cleanup"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.606009    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="124e886b-210d-465f-b6a2-5dcd03b03457" containerName="cleanup"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.606015    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e18350-e8e5-4d56-89e9-dc8dbdf12de8" containerName="cleanup"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.606020    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab8637d-ccad-4855-b7ff-455c744f3da5" containerName="cleanup"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.606026    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="05b7941b-bb75-4e52-8766-7f7eb2f96734" containerName="cleanup"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.606031    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="05b7941b-bb75-4e52-8766-7f7eb2f96734" containerName="cleanup"
Oct 31 14:00:25.606922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.606037    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49565e4-99d7-4a99-a912-047c19c4b8b6" containerName="cleanup"
Oct 31 14:00:25.609721 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.609676    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"
Oct 31 14:00:25.619946 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.619915    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"]
Oct 31 14:00:25.620246 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.620224    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:00:25.620959 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.620935    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:00:25.655453 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.655420    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67c4x\" (UniqueName: \"kubernetes.io/projected/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-kube-api-access-67c4x\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f\" (UID: \"d9298adf-0906-4a7f-9242-0b4b4e0f3a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"
Oct 31 14:00:25.655677 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.655564    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f\" (UID: \"d9298adf-0906-4a7f-9242-0b4b4e0f3a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"
Oct 31 14:00:25.655853 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.655828    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f\" (UID: \"d9298adf-0906-4a7f-9242-0b4b4e0f3a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"
Oct 31 14:00:25.757648 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.757605    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f\" (UID: \"d9298adf-0906-4a7f-9242-0b4b4e0f3a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"
Oct 31 14:00:25.757846 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.757685    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67c4x\" (UniqueName: \"kubernetes.io/projected/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-kube-api-access-67c4x\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f\" (UID: \"d9298adf-0906-4a7f-9242-0b4b4e0f3a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"
Oct 31 14:00:25.757846 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.757775    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f\" (UID: \"d9298adf-0906-4a7f-9242-0b4b4e0f3a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"
Oct 31 14:00:25.758488 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.758459    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f\" (UID: \"d9298adf-0906-4a7f-9242-0b4b4e0f3a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"
Oct 31 14:00:25.760340 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.760321    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f\" (UID: \"d9298adf-0906-4a7f-9242-0b4b4e0f3a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"
Oct 31 14:00:25.772299 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.772218    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67c4x\" (UniqueName: \"kubernetes.io/projected/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-kube-api-access-67c4x\") pod \"maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f\" (UID: \"d9298adf-0906-4a7f-9242-0b4b4e0f3a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"
Oct 31 14:00:25.932209 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:25.932111    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"
Oct 31 14:00:26.224440 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:26.224403    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"]
Oct 31 14:00:26.226174 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:00:26.226142    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9298adf_0906_4a7f_9242_0b4b4e0f3a51.slice/crio-d19dbb087426c5028ad3d0263ea4e0032682d82f63ca231549c336e02b6f6cc5 WatchSource:0}: Error finding container d19dbb087426c5028ad3d0263ea4e0032682d82f63ca231549c336e02b6f6cc5: Status 404 returned error can't find the container with id d19dbb087426c5028ad3d0263ea4e0032682d82f63ca231549c336e02b6f6cc5
Oct 31 14:00:26.391955 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:26.391780    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:26.703538 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:26.703498    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f" event={"ID":"d9298adf-0906-4a7f-9242-0b4b4e0f3a51","Type":"ContainerStarted","Data":"d19dbb087426c5028ad3d0263ea4e0032682d82f63ca231549c336e02b6f6cc5"}
Oct 31 14:00:27.525228 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:27.521240    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:27.713309 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:27.713277    4428 generic.go:356] "Generic (PLEG): container finished" podID="d9298adf-0906-4a7f-9242-0b4b4e0f3a51" containerID="a8dbb5caf7df85f89c368f5a52c0a26c62506949d6522ecea30b510b3dc3d854" exitCode=0
Oct 31 14:00:27.713754 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:27.713362    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f" event={"ID":"d9298adf-0906-4a7f-9242-0b4b4e0f3a51","Type":"ContainerDied","Data":"a8dbb5caf7df85f89c368f5a52c0a26c62506949d6522ecea30b510b3dc3d854"}
Oct 31 14:00:28.402023 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.401988    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 14:00:28.532213 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.532179    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-workload-certs\") pod \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") "
Oct 31 14:00:28.532440 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.532274    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-credential-socket\") pod \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") "
Oct 31 14:00:28.532440 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.532328    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-data\") pod \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") "
Oct 31 14:00:28.532440 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.532357    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-envoy\") pod \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") "
Oct 31 14:00:28.532631 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.532495    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-workload-certs" pod "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" (UID: "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:00:28.532631 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.532504    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istiod-ca-cert\") pod \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") "
Oct 31 14:00:28.532631 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.532615    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-credential-socket" pod "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" (UID: "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:00:28.532785 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.532651    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-data" pod "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" (UID: "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:00:28.532785 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.532670    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-token\" (UniqueName: \"kubernetes.io/projected/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-token\") pod \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") "
Oct 31 14:00:28.532785 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.532707    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-workload-socket\") pod \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") "
Oct 31 14:00:28.532785 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.532760    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-kube-api-access-2z9jv\" (UniqueName: \"kubernetes.io/projected/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-kube-api-access-2z9jv\") pod \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") "
Oct 31 14:00:28.533047 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.532796    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-podinfo\") pod \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\" (UID: \"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d\") "
Oct 31 14:00:28.533047 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.532849    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istiod-ca-cert" pod "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" (UID: "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:28.533047 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.532976    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-workload-socket" pod "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" (UID: "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:00:28.533837 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.533811    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:28.534020 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.533839    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:28.534020 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.533856    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:28.534020 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.533916    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:28.534020 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.533931    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:28.535149 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.535114    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-envoy" pod "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" (UID: "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:00:28.535310 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.535278    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-token" pod "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" (UID: "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:28.535310 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.535280    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-podinfo" pod "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" (UID: "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:00:28.535856 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.535835    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-kube-api-access-2z9jv" pod "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" (UID: "6991ec79-5e5e-4a07-b3c8-dffe08f0d42d"). InnerVolumeSpecName "kube-api-access-2z9jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:28.641953 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.636967    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:28.641953 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.637006    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z9jv\" (UniqueName: \"kubernetes.io/projected/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-kube-api-access-2z9jv\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:28.641953 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.637023    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:28.641953 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.637036    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:28.653146 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.651387    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:28.724497 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.724457    4428 generic.go:356] "Generic (PLEG): container finished" podID="6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" containerID="0a96c1ca088318c212b57afa1adb59adf5f2fd39b61612c5bbb7eab5d1d04d1d" exitCode=0
Oct 31 14:00:28.725055 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.724522    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"
Oct 31 14:00:28.725055 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.724604    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj" event={"ID":"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d","Type":"ContainerDied","Data":"0a96c1ca088318c212b57afa1adb59adf5f2fd39b61612c5bbb7eab5d1d04d1d"}
Oct 31 14:00:28.725055 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.724665    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj" event={"ID":"6991ec79-5e5e-4a07-b3c8-dffe08f0d42d","Type":"ContainerDied","Data":"7056740e8bb7367167c27982d4d3fe28d04e495cb82389603b2a658db4d460a1"}
Oct 31 14:00:28.725055 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.724708    4428 scope.go:122] "RemoveContainer" containerID="0a96c1ca088318c212b57afa1adb59adf5f2fd39b61612c5bbb7eab5d1d04d1d"
Oct 31 14:00:28.749210 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.749183    4428 scope.go:122] "RemoveContainer" containerID="460cfc5b72ecfb55a6b52dbb7b554542b0791c37027e4312fe093a7db81604ef"
Oct 31 14:00:28.769780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.762290    4428 scope.go:122] "RemoveContainer" containerID="0a96c1ca088318c212b57afa1adb59adf5f2fd39b61612c5bbb7eab5d1d04d1d"
Oct 31 14:00:28.769780 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:28.762612    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a96c1ca088318c212b57afa1adb59adf5f2fd39b61612c5bbb7eab5d1d04d1d\": container with ID starting with 0a96c1ca088318c212b57afa1adb59adf5f2fd39b61612c5bbb7eab5d1d04d1d not found: ID does not exist" containerID="0a96c1ca088318c212b57afa1adb59adf5f2fd39b61612c5bbb7eab5d1d04d1d"
Oct 31 14:00:28.769780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.762646    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a96c1ca088318c212b57afa1adb59adf5f2fd39b61612c5bbb7eab5d1d04d1d"} err="failed to get container status \"0a96c1ca088318c212b57afa1adb59adf5f2fd39b61612c5bbb7eab5d1d04d1d\": rpc error: code = NotFound desc = could not find container \"0a96c1ca088318c212b57afa1adb59adf5f2fd39b61612c5bbb7eab5d1d04d1d\": container with ID starting with 0a96c1ca088318c212b57afa1adb59adf5f2fd39b61612c5bbb7eab5d1d04d1d not found: ID does not exist"
Oct 31 14:00:28.769780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.762665    4428 scope.go:122] "RemoveContainer" containerID="460cfc5b72ecfb55a6b52dbb7b554542b0791c37027e4312fe093a7db81604ef"
Oct 31 14:00:28.769780 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:28.762901    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460cfc5b72ecfb55a6b52dbb7b554542b0791c37027e4312fe093a7db81604ef\": container with ID starting with 460cfc5b72ecfb55a6b52dbb7b554542b0791c37027e4312fe093a7db81604ef not found: ID does not exist" containerID="460cfc5b72ecfb55a6b52dbb7b554542b0791c37027e4312fe093a7db81604ef"
Oct 31 14:00:28.769780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.762919    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460cfc5b72ecfb55a6b52dbb7b554542b0791c37027e4312fe093a7db81604ef"} err="failed to get container status \"460cfc5b72ecfb55a6b52dbb7b554542b0791c37027e4312fe093a7db81604ef\": rpc error: code = NotFound desc = could not find container \"460cfc5b72ecfb55a6b52dbb7b554542b0791c37027e4312fe093a7db81604ef\": container with ID starting with 460cfc5b72ecfb55a6b52dbb7b554542b0791c37027e4312fe093a7db81604ef not found: ID does not exist"
Oct 31 14:00:28.769780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.763406    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"]
Oct 31 14:00:28.769780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.769278    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5af6eb-openshift-default-b4c5d69c5-8gxvj"]
Oct 31 14:00:28.904557 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:28.904530    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"
Oct 31 14:00:29.047518 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:29.047476    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-maas-api-xks-ca\") pod \"d9298adf-0906-4a7f-9242-0b4b4e0f3a51\" (UID: \"d9298adf-0906-4a7f-9242-0b4b4e0f3a51\") "
Oct 31 14:00:29.047729 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:29.047627    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-maas-api-openshift-ca\") pod \"d9298adf-0906-4a7f-9242-0b4b4e0f3a51\" (UID: \"d9298adf-0906-4a7f-9242-0b4b4e0f3a51\") "
Oct 31 14:00:29.047729 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:29.047711    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-kube-api-access-67c4x\" (UniqueName: \"kubernetes.io/projected/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-kube-api-access-67c4x\") pod \"d9298adf-0906-4a7f-9242-0b4b4e0f3a51\" (UID: \"d9298adf-0906-4a7f-9242-0b4b4e0f3a51\") "
Oct 31 14:00:29.048048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:29.048020    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-maas-api-openshift-ca" pod "d9298adf-0906-4a7f-9242-0b4b4e0f3a51" (UID: "d9298adf-0906-4a7f-9242-0b4b4e0f3a51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:29.048424 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:29.048398    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:29.050051 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:29.050026    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-maas-api-xks-ca" pod "d9298adf-0906-4a7f-9242-0b4b4e0f3a51" (UID: "d9298adf-0906-4a7f-9242-0b4b4e0f3a51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:00:29.050155 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:29.050086    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-kube-api-access-67c4x" pod "d9298adf-0906-4a7f-9242-0b4b4e0f3a51" (UID: "d9298adf-0906-4a7f-9242-0b4b4e0f3a51"). InnerVolumeSpecName "kube-api-access-67c4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:29.150384 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:29.150296    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67c4x\" (UniqueName: \"kubernetes.io/projected/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-kube-api-access-67c4x\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:29.150384 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:29.150328    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9298adf-0906-4a7f-9242-0b4b4e0f3a51-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:29.537024 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:29.536944    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:29.735685 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:29.735648    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"
Oct 31 14:00:29.735685 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:29.735671    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f" event={"ID":"d9298adf-0906-4a7f-9242-0b4b4e0f3a51","Type":"ContainerDied","Data":"d19dbb087426c5028ad3d0263ea4e0032682d82f63ca231549c336e02b6f6cc5"}
Oct 31 14:00:29.736164 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:29.735724    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d19dbb087426c5028ad3d0263ea4e0032682d82f63ca231549c336e02b6f6cc5"
Oct 31 14:00:30.416492 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:30.416443    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:30.757363 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:30.757326    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" path="/var/lib/kubelet/pods/6991ec79-5e5e-4a07-b3c8-dffe08f0d42d/volumes"
Oct 31 14:00:31.207731 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:31.207632    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:32.270349 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:32.270298    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:33.589891 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:33.588388    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:33.805489 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:33.805453    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:34.008781 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:34.008741    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:34.714082 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:34.714032    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:35.537907 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:35.522514    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:35.679556 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:35.679464    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:35.897799 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:35.897748    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28" podUID="be1f73c9-7c0e-4972-8cae-d8a029d311f3" containerName="main" containerID="cri-o://5a1cba1862752cc95f37a4ec4f0676a6578348d92ec4d09919a708855dcafd88" gracePeriod=45
Oct 31 14:00:35.953931 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:35.953821    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28" podUID="be1f73c9-7c0e-4972-8cae-d8a029d311f3" containerName="main" probeResult="failure" output="Get \"https://10.134.1.11:8000/ready\": dial tcp 10.134.1.11:8000: connect: connection refused"
Oct 31 14:00:36.179429 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.179404    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 14:00:36.243017 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.242984    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-home\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-home\") pod \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") "
Oct 31 14:00:36.243017 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.243027    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-dshm\") pod \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") "
Oct 31 14:00:36.243360 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.243075    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-model-cache\") pod \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") "
Oct 31 14:00:36.243360 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.243173    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be1f73c9-7c0e-4972-8cae-d8a029d311f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/be1f73c9-7c0e-4972-8cae-d8a029d311f3-tls-certs\") pod \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") "
Oct 31 14:00:36.243360 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.243215    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1f73c9-7c0e-4972-8cae-d8a029d311f3-kube-api-access-vc9xw\" (UniqueName: \"kubernetes.io/projected/be1f73c9-7c0e-4972-8cae-d8a029d311f3-kube-api-access-vc9xw\") pod \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") "
Oct 31 14:00:36.243360 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.243321    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-home" pod "be1f73c9-7c0e-4972-8cae-d8a029d311f3" (UID: "be1f73c9-7c0e-4972-8cae-d8a029d311f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:00:36.243360 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.243348    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-tmp-dir\") pod \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\" (UID: \"be1f73c9-7c0e-4972-8cae-d8a029d311f3\") "
Oct 31 14:00:36.244697 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.244193    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:36.244697 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.244262    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-model-cache" pod "be1f73c9-7c0e-4972-8cae-d8a029d311f3" (UID: "be1f73c9-7c0e-4972-8cae-d8a029d311f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:00:36.244697 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.244460    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-tmp-dir" pod "be1f73c9-7c0e-4972-8cae-d8a029d311f3" (UID: "be1f73c9-7c0e-4972-8cae-d8a029d311f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:00:36.248110 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.247996    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-dshm" pod "be1f73c9-7c0e-4972-8cae-d8a029d311f3" (UID: "be1f73c9-7c0e-4972-8cae-d8a029d311f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:00:36.248110 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.248001    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be1f73c9-7c0e-4972-8cae-d8a029d311f3-tls-certs" pod "be1f73c9-7c0e-4972-8cae-d8a029d311f3" (UID: "be1f73c9-7c0e-4972-8cae-d8a029d311f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:00:36.248110 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.248072    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1f73c9-7c0e-4972-8cae-d8a029d311f3-kube-api-access-vc9xw" pod "be1f73c9-7c0e-4972-8cae-d8a029d311f3" (UID: "be1f73c9-7c0e-4972-8cae-d8a029d311f3"). InnerVolumeSpecName "kube-api-access-vc9xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:36.307889 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.307841    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:36.355116 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.351629    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:36.355116 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.351670    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:36.355116 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.351685    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be1f73c9-7c0e-4972-8cae-d8a029d311f3-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:36.355116 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.351700    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc9xw\" (UniqueName: \"kubernetes.io/projected/be1f73c9-7c0e-4972-8cae-d8a029d311f3-kube-api-access-vc9xw\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:36.355116 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.351717    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be1f73c9-7c0e-4972-8cae-d8a029d311f3-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:36.825557 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.825522    4428 generic.go:356] "Generic (PLEG): container finished" podID="be1f73c9-7c0e-4972-8cae-d8a029d311f3" containerID="5a1cba1862752cc95f37a4ec4f0676a6578348d92ec4d09919a708855dcafd88" exitCode=0
Oct 31 14:00:36.825777 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.825571    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"
Oct 31 14:00:36.825777 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.825599    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28" event={"ID":"be1f73c9-7c0e-4972-8cae-d8a029d311f3","Type":"ContainerDied","Data":"5a1cba1862752cc95f37a4ec4f0676a6578348d92ec4d09919a708855dcafd88"}
Oct 31 14:00:36.825777 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.825652    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28" event={"ID":"be1f73c9-7c0e-4972-8cae-d8a029d311f3","Type":"ContainerDied","Data":"719ce59196d306d4f21c8d51f8bb0b94c92a2edf08f7108462d2c7d611c0e184"}
Oct 31 14:00:36.825777 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.825677    4428 scope.go:122] "RemoveContainer" containerID="5a1cba1862752cc95f37a4ec4f0676a6578348d92ec4d09919a708855dcafd88"
Oct 31 14:00:36.838786 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.838760    4428 scope.go:122] "RemoveContainer" containerID="5a1cba1862752cc95f37a4ec4f0676a6578348d92ec4d09919a708855dcafd88"
Oct 31 14:00:36.839160 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:36.839133    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a1cba1862752cc95f37a4ec4f0676a6578348d92ec4d09919a708855dcafd88\": container with ID starting with 5a1cba1862752cc95f37a4ec4f0676a6578348d92ec4d09919a708855dcafd88 not found: ID does not exist" containerID="5a1cba1862752cc95f37a4ec4f0676a6578348d92ec4d09919a708855dcafd88"
Oct 31 14:00:36.839254 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.839168    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1cba1862752cc95f37a4ec4f0676a6578348d92ec4d09919a708855dcafd88"} err="failed to get container status \"5a1cba1862752cc95f37a4ec4f0676a6578348d92ec4d09919a708855dcafd88\": rpc error: code = NotFound desc = could not find container \"5a1cba1862752cc95f37a4ec4f0676a6578348d92ec4d09919a708855dcafd88\": container with ID starting with 5a1cba1862752cc95f37a4ec4f0676a6578348d92ec4d09919a708855dcafd88 not found: ID does not exist"
Oct 31 14:00:36.857508 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.857447    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"]
Oct 31 14:00:36.861444 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:36.861416    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a81979/ipp-route-a81979-kserve-7bd7bcc49-hsd28"]
Oct 31 14:00:37.168960 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:37.168764    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:37.980895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:37.974532    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:38.769563 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:38.769531    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be1f73c9-7c0e-4972-8cae-d8a029d311f3" path="/var/lib/kubelet/pods/be1f73c9-7c0e-4972-8cae-d8a029d311f3/volumes"
Oct 31 14:00:38.791292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:38.791264    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:39.649944 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:39.649816    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:40.501994 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:40.501952    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:41.314898 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:41.313860    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:42.065416 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:42.065382    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:42.838939 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:42.833040    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:43.843960 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:43.836685    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:44.507924 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.503655    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"]
Oct 31 14:00:44.507924 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.503937    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv" podUID="4059a5cf-e441-4c84-9f57-e237f45a853f" containerName="maas-api" containerID="cri-o://e1a3b8b1c64f7cc6967cfde0e4bb28b4dbea8a06bcf05613d982345245a4585d" gracePeriod=30
Oct 31 14:00:44.657894 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.654830    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:44.670893 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.669471    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"]
Oct 31 14:00:44.670893 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.669759    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96" podUID="cf34172e-3804-4277-aeab-900beeaeefe7" containerName="payload-processing" containerID="cri-o://4ab0e5eb28a04084a4da3041e0a04ff0888a889f8126fdba35d74d1f955ee9bc" gracePeriod=30
Oct 31 14:00:44.695012 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.686341    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"]
Oct 31 14:00:44.695012 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.686601    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d" podUID="fdb433fc-0251-4148-b9b3-ec57bf90be3a" containerName="payload-pre-processing" containerID="cri-o://a50763f471afb7baa9cbf848e97357119d92b7c6375175301e37e668ee0a6dff" gracePeriod=30
Oct 31 14:00:44.791645 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.791601    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"]
Oct 31 14:00:44.792640 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.792597    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q" podUID="a9b22227-029e-4dff-8530-b5afcdbf58dd" containerName="maas-api" containerID="cri-o://6e6912ed97f019cb3c0a1c78641d8af0f1df63b5b875d1ec56fee03a18ac9f43" gracePeriod=30
Oct 31 14:00:44.821645 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.821605    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:44.892762 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.892721    4428 generic.go:356] "Generic (PLEG): container finished" podID="fdb433fc-0251-4148-b9b3-ec57bf90be3a" containerID="a50763f471afb7baa9cbf848e97357119d92b7c6375175301e37e668ee0a6dff" exitCode=0
Oct 31 14:00:44.893240 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.892791    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d" event={"ID":"fdb433fc-0251-4148-b9b3-ec57bf90be3a","Type":"ContainerDied","Data":"a50763f471afb7baa9cbf848e97357119d92b7c6375175301e37e668ee0a6dff"}
Oct 31 14:00:44.894985 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.894953    4428 generic.go:356] "Generic (PLEG): container finished" podID="cf34172e-3804-4277-aeab-900beeaeefe7" containerID="4ab0e5eb28a04084a4da3041e0a04ff0888a889f8126fdba35d74d1f955ee9bc" exitCode=0
Oct 31 14:00:44.895135 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.894999    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96" event={"ID":"cf34172e-3804-4277-aeab-900beeaeefe7","Type":"ContainerDied","Data":"4ab0e5eb28a04084a4da3041e0a04ff0888a889f8126fdba35d74d1f955ee9bc"}
Oct 31 14:00:44.897234 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.897211    4428 generic.go:356] "Generic (PLEG): container finished" podID="4059a5cf-e441-4c84-9f57-e237f45a853f" containerID="e1a3b8b1c64f7cc6967cfde0e4bb28b4dbea8a06bcf05613d982345245a4585d" exitCode=0
Oct 31 14:00:44.897354 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.897239    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv" event={"ID":"4059a5cf-e441-4c84-9f57-e237f45a853f","Type":"ContainerDied","Data":"e1a3b8b1c64f7cc6967cfde0e4bb28b4dbea8a06bcf05613d982345245a4585d"}
Oct 31 14:00:44.949266 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.949215    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"]
Oct 31 14:00:44.953826 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.953800    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 14:00:44.953947 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.953886    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9f9103-6qn4f"]
Oct 31 14:00:45.012309 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:44.995904    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:45.028896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.027543    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4059a5cf-e441-4c84-9f57-e237f45a853f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4059a5cf-e441-4c84-9f57-e237f45a853f-openshift-service-ca\") pod \"4059a5cf-e441-4c84-9f57-e237f45a853f\" (UID: \"4059a5cf-e441-4c84-9f57-e237f45a853f\") "
Oct 31 14:00:45.028896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.027635    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4059a5cf-e441-4c84-9f57-e237f45a853f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4059a5cf-e441-4c84-9f57-e237f45a853f-metrics-certs\") pod \"4059a5cf-e441-4c84-9f57-e237f45a853f\" (UID: \"4059a5cf-e441-4c84-9f57-e237f45a853f\") "
Oct 31 14:00:45.028896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.027735    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4059a5cf-e441-4c84-9f57-e237f45a853f-kube-api-access-cvhv8\" (UniqueName: \"kubernetes.io/projected/4059a5cf-e441-4c84-9f57-e237f45a853f-kube-api-access-cvhv8\") pod \"4059a5cf-e441-4c84-9f57-e237f45a853f\" (UID: \"4059a5cf-e441-4c84-9f57-e237f45a853f\") "
Oct 31 14:00:45.028896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.027777    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4059a5cf-e441-4c84-9f57-e237f45a853f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4059a5cf-e441-4c84-9f57-e237f45a853f-maas-api-tls\") pod \"4059a5cf-e441-4c84-9f57-e237f45a853f\" (UID: \"4059a5cf-e441-4c84-9f57-e237f45a853f\") "
Oct 31 14:00:45.031066 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.031019    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4059a5cf-e441-4c84-9f57-e237f45a853f-openshift-service-ca" pod "4059a5cf-e441-4c84-9f57-e237f45a853f" (UID: "4059a5cf-e441-4c84-9f57-e237f45a853f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:45.096155 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.096088    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4059a5cf-e441-4c84-9f57-e237f45a853f-kube-api-access-cvhv8" pod "4059a5cf-e441-4c84-9f57-e237f45a853f" (UID: "4059a5cf-e441-4c84-9f57-e237f45a853f"). InnerVolumeSpecName "kube-api-access-cvhv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:45.107646 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.107575    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4059a5cf-e441-4c84-9f57-e237f45a853f-maas-api-tls" pod "4059a5cf-e441-4c84-9f57-e237f45a853f" (UID: "4059a5cf-e441-4c84-9f57-e237f45a853f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:00:45.131489 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.131387    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4059a5cf-e441-4c84-9f57-e237f45a853f-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.131489 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.131428    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvhv8\" (UniqueName: \"kubernetes.io/projected/4059a5cf-e441-4c84-9f57-e237f45a853f-kube-api-access-cvhv8\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.131489 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.131446    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4059a5cf-e441-4c84-9f57-e237f45a853f-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.191949 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.176901    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:45.245319 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.245249    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4059a5cf-e441-4c84-9f57-e237f45a853f-metrics-certs" pod "4059a5cf-e441-4c84-9f57-e237f45a853f" (UID: "4059a5cf-e441-4c84-9f57-e237f45a853f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:00:45.251500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.251440    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4059a5cf-e441-4c84-9f57-e237f45a853f-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.287550 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.287126    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"
Oct 31 14:00:45.346115 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.346080    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"
Oct 31 14:00:45.403265 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.403228    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:45.415646 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.415619    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 14:00:45.454353 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.454311    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf34172e-3804-4277-aeab-900beeaeefe7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf34172e-3804-4277-aeab-900beeaeefe7-plugins-config-volume\") pod \"cf34172e-3804-4277-aeab-900beeaeefe7\" (UID: \"cf34172e-3804-4277-aeab-900beeaeefe7\") "
Oct 31 14:00:45.454599 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.454464    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb433fc-0251-4148-b9b3-ec57bf90be3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb433fc-0251-4148-b9b3-ec57bf90be3a-plugins-config-volume\") pod \"fdb433fc-0251-4148-b9b3-ec57bf90be3a\" (UID: \"fdb433fc-0251-4148-b9b3-ec57bf90be3a\") "
Oct 31 14:00:45.454731 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.454691    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf34172e-3804-4277-aeab-900beeaeefe7-kube-api-access-j79w9\" (UniqueName: \"kubernetes.io/projected/cf34172e-3804-4277-aeab-900beeaeefe7-kube-api-access-j79w9\") pod \"cf34172e-3804-4277-aeab-900beeaeefe7\" (UID: \"cf34172e-3804-4277-aeab-900beeaeefe7\") "
Oct 31 14:00:45.454731 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.454720    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb433fc-0251-4148-b9b3-ec57bf90be3a-kube-api-access-4rcqj\" (UniqueName: \"kubernetes.io/projected/fdb433fc-0251-4148-b9b3-ec57bf90be3a-kube-api-access-4rcqj\") pod \"fdb433fc-0251-4148-b9b3-ec57bf90be3a\" (UID: \"fdb433fc-0251-4148-b9b3-ec57bf90be3a\") "
Oct 31 14:00:45.454922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.454895    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf34172e-3804-4277-aeab-900beeaeefe7-plugins-config-volume" pod "cf34172e-3804-4277-aeab-900beeaeefe7" (UID: "cf34172e-3804-4277-aeab-900beeaeefe7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:45.455019 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.454965    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb433fc-0251-4148-b9b3-ec57bf90be3a-plugins-config-volume" pod "fdb433fc-0251-4148-b9b3-ec57bf90be3a" (UID: "fdb433fc-0251-4148-b9b3-ec57bf90be3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:45.455521 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.455475    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf34172e-3804-4277-aeab-900beeaeefe7-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.455521 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.455504    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb433fc-0251-4148-b9b3-ec57bf90be3a-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.457844 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.457817    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb433fc-0251-4148-b9b3-ec57bf90be3a-kube-api-access-4rcqj" pod "fdb433fc-0251-4148-b9b3-ec57bf90be3a" (UID: "fdb433fc-0251-4148-b9b3-ec57bf90be3a"). InnerVolumeSpecName "kube-api-access-4rcqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:45.458009 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.457852    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf34172e-3804-4277-aeab-900beeaeefe7-kube-api-access-j79w9" pod "cf34172e-3804-4277-aeab-900beeaeefe7" (UID: "cf34172e-3804-4277-aeab-900beeaeefe7"). InnerVolumeSpecName "kube-api-access-j79w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:45.556764 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.556679    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9b22227-029e-4dff-8530-b5afcdbf58dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9b22227-029e-4dff-8530-b5afcdbf58dd-maas-api-tls\") pod \"a9b22227-029e-4dff-8530-b5afcdbf58dd\" (UID: \"a9b22227-029e-4dff-8530-b5afcdbf58dd\") "
Oct 31 14:00:45.556932 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.556777    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9b22227-029e-4dff-8530-b5afcdbf58dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9b22227-029e-4dff-8530-b5afcdbf58dd-metrics-certs\") pod \"a9b22227-029e-4dff-8530-b5afcdbf58dd\" (UID: \"a9b22227-029e-4dff-8530-b5afcdbf58dd\") "
Oct 31 14:00:45.556932 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.556850    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b22227-029e-4dff-8530-b5afcdbf58dd-kube-api-access-dlf87\" (UniqueName: \"kubernetes.io/projected/a9b22227-029e-4dff-8530-b5afcdbf58dd-kube-api-access-dlf87\") pod \"a9b22227-029e-4dff-8530-b5afcdbf58dd\" (UID: \"a9b22227-029e-4dff-8530-b5afcdbf58dd\") "
Oct 31 14:00:45.557015 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.557000    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b22227-029e-4dff-8530-b5afcdbf58dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b22227-029e-4dff-8530-b5afcdbf58dd-openshift-service-ca\") pod \"a9b22227-029e-4dff-8530-b5afcdbf58dd\" (UID: \"a9b22227-029e-4dff-8530-b5afcdbf58dd\") "
Oct 31 14:00:45.557442 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.557356    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b22227-029e-4dff-8530-b5afcdbf58dd-openshift-service-ca" pod "a9b22227-029e-4dff-8530-b5afcdbf58dd" (UID: "a9b22227-029e-4dff-8530-b5afcdbf58dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:45.557678 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.557592    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j79w9\" (UniqueName: \"kubernetes.io/projected/cf34172e-3804-4277-aeab-900beeaeefe7-kube-api-access-j79w9\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.557678 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.557612    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rcqj\" (UniqueName: \"kubernetes.io/projected/fdb433fc-0251-4148-b9b3-ec57bf90be3a-kube-api-access-4rcqj\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.557678 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.557629    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b22227-029e-4dff-8530-b5afcdbf58dd-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.559226 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.559201    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b22227-029e-4dff-8530-b5afcdbf58dd-maas-api-tls" pod "a9b22227-029e-4dff-8530-b5afcdbf58dd" (UID: "a9b22227-029e-4dff-8530-b5afcdbf58dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:00:45.559333 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.559222    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b22227-029e-4dff-8530-b5afcdbf58dd-metrics-certs" pod "a9b22227-029e-4dff-8530-b5afcdbf58dd" (UID: "a9b22227-029e-4dff-8530-b5afcdbf58dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:00:45.559333 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.559308    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b22227-029e-4dff-8530-b5afcdbf58dd-kube-api-access-dlf87" pod "a9b22227-029e-4dff-8530-b5afcdbf58dd" (UID: "a9b22227-029e-4dff-8530-b5afcdbf58dd"). InnerVolumeSpecName "kube-api-access-dlf87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:45.561242 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.561217    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:45.667965 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.661182    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9b22227-029e-4dff-8530-b5afcdbf58dd-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.667965 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.661228    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlf87\" (UniqueName: \"kubernetes.io/projected/a9b22227-029e-4dff-8530-b5afcdbf58dd-kube-api-access-dlf87\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.667965 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.661245    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9b22227-029e-4dff-8530-b5afcdbf58dd-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:45.726106 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.726063    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:45.907083 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.906989    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"
Oct 31 14:00:45.907083 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.907014    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv" event={"ID":"4059a5cf-e441-4c84-9f57-e237f45a853f","Type":"ContainerDied","Data":"9e937ea0f773cf7d60b0d550f4bfcd7ae5a5f25c77de3ac542952e29aea9c765"}
Oct 31 14:00:45.907083 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.907077    4428 scope.go:122] "RemoveContainer" containerID="e1a3b8b1c64f7cc6967cfde0e4bb28b4dbea8a06bcf05613d982345245a4585d"
Oct 31 14:00:45.908965 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.908945    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"
Oct 31 14:00:45.909110 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.908945    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d" event={"ID":"fdb433fc-0251-4148-b9b3-ec57bf90be3a","Type":"ContainerDied","Data":"81397089ee03c1a571a372137cedf14a1172b06be10503e40e1a0e47793c0d5c"}
Oct 31 14:00:45.911296 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.911235    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96" event={"ID":"cf34172e-3804-4277-aeab-900beeaeefe7","Type":"ContainerDied","Data":"3a51eeb86fcda9e63f9fdcc56f4a83eab7cecde9b6c9e9159d1f2aede58103df"}
Oct 31 14:00:45.911436 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.911252    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"
Oct 31 14:00:45.913351 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.913241    4428 generic.go:356] "Generic (PLEG): container finished" podID="a9b22227-029e-4dff-8530-b5afcdbf58dd" containerID="6e6912ed97f019cb3c0a1c78641d8af0f1df63b5b875d1ec56fee03a18ac9f43" exitCode=0
Oct 31 14:00:45.913351 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.913290    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"
Oct 31 14:00:45.913351 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.913298    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q" event={"ID":"a9b22227-029e-4dff-8530-b5afcdbf58dd","Type":"ContainerDied","Data":"6e6912ed97f019cb3c0a1c78641d8af0f1df63b5b875d1ec56fee03a18ac9f43"}
Oct 31 14:00:45.913351 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.913352    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q" event={"ID":"a9b22227-029e-4dff-8530-b5afcdbf58dd","Type":"ContainerDied","Data":"bef487076e2143d9a65f2e27d47f8095f99ea485d9804d09088f0cf202deddb8"}
Oct 31 14:00:45.928970 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.928933    4428 scope.go:122] "RemoveContainer" containerID="a50763f471afb7baa9cbf848e97357119d92b7c6375175301e37e668ee0a6dff"
Oct 31 14:00:45.939558 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.939534    4428 scope.go:122] "RemoveContainer" containerID="4ab0e5eb28a04084a4da3041e0a04ff0888a889f8126fdba35d74d1f955ee9bc"
Oct 31 14:00:45.947649 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.947622    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"]
Oct 31 14:00:45.953357 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.953327    4428 scope.go:122] "RemoveContainer" containerID="6e6912ed97f019cb3c0a1c78641d8af0f1df63b5b875d1ec56fee03a18ac9f43"
Oct 31 14:00:45.955074 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.955052    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f9103-6bf8f4c9cf-vhf9d"]
Oct 31 14:00:45.964399 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.964372    4428 scope.go:122] "RemoveContainer" containerID="6e6912ed97f019cb3c0a1c78641d8af0f1df63b5b875d1ec56fee03a18ac9f43"
Oct 31 14:00:45.964732 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:45.964703    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6912ed97f019cb3c0a1c78641d8af0f1df63b5b875d1ec56fee03a18ac9f43\": container with ID starting with 6e6912ed97f019cb3c0a1c78641d8af0f1df63b5b875d1ec56fee03a18ac9f43 not found: ID does not exist" containerID="6e6912ed97f019cb3c0a1c78641d8af0f1df63b5b875d1ec56fee03a18ac9f43"
Oct 31 14:00:45.964798 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.964746    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6912ed97f019cb3c0a1c78641d8af0f1df63b5b875d1ec56fee03a18ac9f43"} err="failed to get container status \"6e6912ed97f019cb3c0a1c78641d8af0f1df63b5b875d1ec56fee03a18ac9f43\": rpc error: code = NotFound desc = could not find container \"6e6912ed97f019cb3c0a1c78641d8af0f1df63b5b875d1ec56fee03a18ac9f43\": container with ID starting with 6e6912ed97f019cb3c0a1c78641d8af0f1df63b5b875d1ec56fee03a18ac9f43 not found: ID does not exist"
Oct 31 14:00:45.968250 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.968223    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"]
Oct 31 14:00:45.972300 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.972277    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f9103-7449ffc9c6-xtbtv"]
Oct 31 14:00:45.984828 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.984800    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"]
Oct 31 14:00:45.988791 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:45.988765    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-97927b-88c6c9-g7r9q"]
Oct 31 14:00:46.004026 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:46.003998    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"]
Oct 31 14:00:46.007695 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:46.007670    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f9103-7b88cf99d6-t4t96"]
Oct 31 14:00:46.072784 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:46.072755    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:46.232695 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:46.232654    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:46.757507 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:46.757473    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4059a5cf-e441-4c84-9f57-e237f45a853f" path="/var/lib/kubelet/pods/4059a5cf-e441-4c84-9f57-e237f45a853f/volumes"
Oct 31 14:00:46.758068 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:46.758051    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b22227-029e-4dff-8530-b5afcdbf58dd" path="/var/lib/kubelet/pods/a9b22227-029e-4dff-8530-b5afcdbf58dd/volumes"
Oct 31 14:00:46.758438 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:46.758425    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf34172e-3804-4277-aeab-900beeaeefe7" path="/var/lib/kubelet/pods/cf34172e-3804-4277-aeab-900beeaeefe7/volumes"
Oct 31 14:00:46.759168 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:46.759152    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9298adf-0906-4a7f-9242-0b4b4e0f3a51" path="/var/lib/kubelet/pods/d9298adf-0906-4a7f-9242-0b4b4e0f3a51/volumes"
Oct 31 14:00:46.759606 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:46.759591    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb433fc-0251-4148-b9b3-ec57bf90be3a" path="/var/lib/kubelet/pods/fdb433fc-0251-4148-b9b3-ec57bf90be3a/volumes"
Oct 31 14:00:46.814913 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:46.813789    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:47.821240 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:47.821204    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:48.743103 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:00:48.743067    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:00:48.831811 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:48.831488    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:49.599055 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.596210    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:49.637228 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.637188    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"]
Oct 31 14:00:49.637969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.637944    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9ab8637d-ccad-4855-b7ff-455c744f3da5" containerName="cleanup"
Oct 31 14:00:49.637969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.637967    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab8637d-ccad-4855-b7ff-455c744f3da5" containerName="cleanup"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.637999    4428 cpu_manager.go:414] "containerMap: removing container" podUID="05b7941b-bb75-4e52-8766-7f7eb2f96734" containerName="cleanup"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638004    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05b7941b-bb75-4e52-8766-7f7eb2f96734" containerName="cleanup"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638015    4428 cpu_manager.go:414] "containerMap: removing container" podUID="a9b22227-029e-4dff-8530-b5afcdbf58dd" containerName="maas-api"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638020    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b22227-029e-4dff-8530-b5afcdbf58dd" containerName="maas-api"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638031    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d9298adf-0906-4a7f-9242-0b4b4e0f3a51" containerName="revoke-keys"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638039    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9298adf-0906-4a7f-9242-0b4b4e0f3a51" containerName="revoke-keys"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638054    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4059a5cf-e441-4c84-9f57-e237f45a853f" containerName="maas-api"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638059    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4059a5cf-e441-4c84-9f57-e237f45a853f" containerName="maas-api"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638070    4428 cpu_manager.go:414] "containerMap: removing container" podUID="6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" containerName="istio-proxy"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638075    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" containerName="istio-proxy"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638085    4428 cpu_manager.go:414] "containerMap: removing container" podUID="cf34172e-3804-4277-aeab-900beeaeefe7" containerName="payload-processing"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638089    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf34172e-3804-4277-aeab-900beeaeefe7" containerName="payload-processing"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638113    4428 cpu_manager.go:414] "containerMap: removing container" podUID="be1f73c9-7c0e-4972-8cae-d8a029d311f3" containerName="main"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638122    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1f73c9-7c0e-4972-8cae-d8a029d311f3" containerName="main"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638139    4428 cpu_manager.go:414] "containerMap: removing container" podUID="fdb433fc-0251-4148-b9b3-ec57bf90be3a" containerName="payload-pre-processing"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638144    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb433fc-0251-4148-b9b3-ec57bf90be3a" containerName="payload-pre-processing"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638154    4428 cpu_manager.go:414] "containerMap: removing container" podUID="6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" containerName="istio-proxy"
Oct 31 14:00:49.638214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638158    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" containerName="istio-proxy"
Oct 31 14:00:49.639247 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638281    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="124e886b-210d-465f-b6a2-5dcd03b03457" containerName="cleanup"
Oct 31 14:00:49.639247 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638294    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb433fc-0251-4148-b9b3-ec57bf90be3a" containerName="payload-pre-processing"
Oct 31 14:00:49.639247 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638301    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" containerName="istio-proxy"
Oct 31 14:00:49.639247 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638307    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="05b7941b-bb75-4e52-8766-7f7eb2f96734" containerName="cleanup"
Oct 31 14:00:49.639247 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638316    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b22227-029e-4dff-8530-b5afcdbf58dd" containerName="maas-api"
Oct 31 14:00:49.639247 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638323    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9298adf-0906-4a7f-9242-0b4b4e0f3a51" containerName="revoke-keys"
Oct 31 14:00:49.639247 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638328    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4059a5cf-e441-4c84-9f57-e237f45a853f" containerName="maas-api"
Oct 31 14:00:49.639247 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638335    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49565e4-99d7-4a99-a912-047c19c4b8b6" containerName="cleanup"
Oct 31 14:00:49.639247 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638343    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf34172e-3804-4277-aeab-900beeaeefe7" containerName="payload-processing"
Oct 31 14:00:49.639247 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638352    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fee8a48-bc4e-4df9-b4db-354d2f577fd0" containerName="cleanup"
Oct 31 14:00:49.639247 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.638361    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1f73c9-7c0e-4972-8cae-d8a029d311f3" containerName="main"
Oct 31 14:00:49.643795 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.643773    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"
Oct 31 14:00:49.649234 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.649206    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:00:49.649234 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.649225    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:00:49.653286 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.653263    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"]
Oct 31 14:00:49.733794 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.733756    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f431fba-c1ad-4520-899e-774be2a5d007-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b0a7360-4h5d2\" (UID: \"4f431fba-c1ad-4520-899e-774be2a5d007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"
Oct 31 14:00:49.734007 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.733824    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f431fba-c1ad-4520-899e-774be2a5d007-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b0a7360-4h5d2\" (UID: \"4f431fba-c1ad-4520-899e-774be2a5d007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"
Oct 31 14:00:49.734083 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.734058    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc9tl\" (UniqueName: \"kubernetes.io/projected/4f431fba-c1ad-4520-899e-774be2a5d007-kube-api-access-bc9tl\") pod \"maas-api-revoke-sub-9b0a7360-4h5d2\" (UID: \"4f431fba-c1ad-4520-899e-774be2a5d007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"
Oct 31 14:00:49.836082 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.836033    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f431fba-c1ad-4520-899e-774be2a5d007-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b0a7360-4h5d2\" (UID: \"4f431fba-c1ad-4520-899e-774be2a5d007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"
Oct 31 14:00:49.836514 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.836132    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f431fba-c1ad-4520-899e-774be2a5d007-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b0a7360-4h5d2\" (UID: \"4f431fba-c1ad-4520-899e-774be2a5d007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"
Oct 31 14:00:49.836514 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.836243    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9tl\" (UniqueName: \"kubernetes.io/projected/4f431fba-c1ad-4520-899e-774be2a5d007-kube-api-access-bc9tl\") pod \"maas-api-revoke-sub-9b0a7360-4h5d2\" (UID: \"4f431fba-c1ad-4520-899e-774be2a5d007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"
Oct 31 14:00:49.836742 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.836721    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f431fba-c1ad-4520-899e-774be2a5d007-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b0a7360-4h5d2\" (UID: \"4f431fba-c1ad-4520-899e-774be2a5d007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"
Oct 31 14:00:49.838668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.838650    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f431fba-c1ad-4520-899e-774be2a5d007-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b0a7360-4h5d2\" (UID: \"4f431fba-c1ad-4520-899e-774be2a5d007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"
Oct 31 14:00:49.869692 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.869623    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc9tl\" (UniqueName: \"kubernetes.io/projected/4f431fba-c1ad-4520-899e-774be2a5d007-kube-api-access-bc9tl\") pod \"maas-api-revoke-sub-9b0a7360-4h5d2\" (UID: \"4f431fba-c1ad-4520-899e-774be2a5d007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"
Oct 31 14:00:49.962951 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:49.962915    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"
Oct 31 14:00:50.258374 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:50.258346    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"]
Oct 31 14:00:50.259616 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:00:50.259594    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f431fba_c1ad_4520_899e_774be2a5d007.slice/crio-ae57c2dc8c95fc37d1e74df221a00471756ae7c4230907cf6ccd6a1009c3f4ba WatchSource:0}: Error finding container ae57c2dc8c95fc37d1e74df221a00471756ae7c4230907cf6ccd6a1009c3f4ba: Status 404 returned error can't find the container with id ae57c2dc8c95fc37d1e74df221a00471756ae7c4230907cf6ccd6a1009c3f4ba
Oct 31 14:00:50.476679 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:50.476634    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:50.972666 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:50.972615    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2" event={"ID":"4f431fba-c1ad-4520-899e-774be2a5d007","Type":"ContainerStarted","Data":"d9eea432b31b7e7ed40d0b83026141bf68772cd61acb5e2c6e9ebea6bcc18975"}
Oct 31 14:00:50.973472 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:50.972696    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2" event={"ID":"4f431fba-c1ad-4520-899e-774be2a5d007","Type":"ContainerStarted","Data":"ae57c2dc8c95fc37d1e74df221a00471756ae7c4230907cf6ccd6a1009c3f4ba"}
Oct 31 14:00:50.996584 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:50.996518    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2" podStartSLOduration=1.996502375 podStartE2EDuration="1.996502375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:50.99259858 +0000 UTC m=+3448.390947207" watchObservedRunningTime="2026-10-31 14:00:50.996502375 +0000 UTC m=+3448.394850993"
Oct 31 14:00:51.275880 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:51.275764    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:51.981909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:51.981855    4428 generic.go:356] "Generic (PLEG): container finished" podID="4f431fba-c1ad-4520-899e-774be2a5d007" containerID="d9eea432b31b7e7ed40d0b83026141bf68772cd61acb5e2c6e9ebea6bcc18975" exitCode=0
Oct 31 14:00:51.982387 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:51.981930    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2" event={"ID":"4f431fba-c1ad-4520-899e-774be2a5d007","Type":"ContainerDied","Data":"d9eea432b31b7e7ed40d0b83026141bf68772cd61acb5e2c6e9ebea6bcc18975"}
Oct 31 14:00:52.122059 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:52.121605    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:53.031372 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:53.031336    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:53.214021 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:53.213992    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"
Oct 31 14:00:53.257653 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:53.257609    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:53.297226 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:53.297140    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f431fba-c1ad-4520-899e-774be2a5d007-kube-api-access-bc9tl\" (UniqueName: \"kubernetes.io/projected/4f431fba-c1ad-4520-899e-774be2a5d007-kube-api-access-bc9tl\") pod \"4f431fba-c1ad-4520-899e-774be2a5d007\" (UID: \"4f431fba-c1ad-4520-899e-774be2a5d007\") "
Oct 31 14:00:53.297226 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:53.297208    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f431fba-c1ad-4520-899e-774be2a5d007-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f431fba-c1ad-4520-899e-774be2a5d007-maas-api-openshift-ca\") pod \"4f431fba-c1ad-4520-899e-774be2a5d007\" (UID: \"4f431fba-c1ad-4520-899e-774be2a5d007\") "
Oct 31 14:00:53.297430 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:53.297303    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f431fba-c1ad-4520-899e-774be2a5d007-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f431fba-c1ad-4520-899e-774be2a5d007-maas-api-xks-ca\") pod \"4f431fba-c1ad-4520-899e-774be2a5d007\" (UID: \"4f431fba-c1ad-4520-899e-774be2a5d007\") "
Oct 31 14:00:53.297664 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:53.297626    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f431fba-c1ad-4520-899e-774be2a5d007-maas-api-openshift-ca" pod "4f431fba-c1ad-4520-899e-774be2a5d007" (UID: "4f431fba-c1ad-4520-899e-774be2a5d007"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:53.297761 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:53.297740    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f431fba-c1ad-4520-899e-774be2a5d007-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:53.299570 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:53.299542    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f431fba-c1ad-4520-899e-774be2a5d007-maas-api-xks-ca" pod "4f431fba-c1ad-4520-899e-774be2a5d007" (UID: "4f431fba-c1ad-4520-899e-774be2a5d007"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:00:53.299677 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:53.299608    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f431fba-c1ad-4520-899e-774be2a5d007-kube-api-access-bc9tl" pod "4f431fba-c1ad-4520-899e-774be2a5d007" (UID: "4f431fba-c1ad-4520-899e-774be2a5d007"). InnerVolumeSpecName "kube-api-access-bc9tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:53.399372 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:53.399310    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc9tl\" (UniqueName: \"kubernetes.io/projected/4f431fba-c1ad-4520-899e-774be2a5d007-kube-api-access-bc9tl\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:53.399372 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:53.399347    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f431fba-c1ad-4520-899e-774be2a5d007-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:53.493118 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:53.493076    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:53.713068 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:53.690479    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:54.045729 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.045618    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2" event={"ID":"4f431fba-c1ad-4520-899e-774be2a5d007","Type":"ContainerDied","Data":"ae57c2dc8c95fc37d1e74df221a00471756ae7c4230907cf6ccd6a1009c3f4ba"}
Oct 31 14:00:54.045729 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.045686    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae57c2dc8c95fc37d1e74df221a00471756ae7c4230907cf6ccd6a1009c3f4ba"
Oct 31 14:00:54.046333 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.045759    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"
Oct 31 14:00:54.128578 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.128545    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:54.157788 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.157247    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"]
Oct 31 14:00:54.159343 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.159313    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4f431fba-c1ad-4520-899e-774be2a5d007" containerName="revoke-keys"
Oct 31 14:00:54.159599 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.159567    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f431fba-c1ad-4520-899e-774be2a5d007" containerName="revoke-keys"
Oct 31 14:00:54.160230 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.160199    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f431fba-c1ad-4520-899e-774be2a5d007" containerName="revoke-keys"
Oct 31 14:00:54.160342 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.160241    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6991ec79-5e5e-4a07-b3c8-dffe08f0d42d" containerName="istio-proxy"
Oct 31 14:00:54.172759 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.172724    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"
Oct 31 14:00:54.178761 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.178733    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:00:54.180910 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.179528    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:00:54.180910 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.180030    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"]
Oct 31 14:00:54.226794 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.226749    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsqld\" (UniqueName: \"kubernetes.io/projected/11c4087d-1e63-4c89-b69b-52cfd37db57b-kube-api-access-tsqld\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9\" (UID: \"11c4087d-1e63-4c89-b69b-52cfd37db57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"
Oct 31 14:00:54.226979 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.226898    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c4087d-1e63-4c89-b69b-52cfd37db57b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9\" (UID: \"11c4087d-1e63-4c89-b69b-52cfd37db57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"
Oct 31 14:00:54.227231 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.227205    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c4087d-1e63-4c89-b69b-52cfd37db57b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9\" (UID: \"11c4087d-1e63-4c89-b69b-52cfd37db57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"
Oct 31 14:00:54.298505 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.298401    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:54.328800 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.328763    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c4087d-1e63-4c89-b69b-52cfd37db57b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9\" (UID: \"11c4087d-1e63-4c89-b69b-52cfd37db57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"
Oct 31 14:00:54.329042 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.328883    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsqld\" (UniqueName: \"kubernetes.io/projected/11c4087d-1e63-4c89-b69b-52cfd37db57b-kube-api-access-tsqld\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9\" (UID: \"11c4087d-1e63-4c89-b69b-52cfd37db57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"
Oct 31 14:00:54.329042 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.328941    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c4087d-1e63-4c89-b69b-52cfd37db57b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9\" (UID: \"11c4087d-1e63-4c89-b69b-52cfd37db57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"
Oct 31 14:00:54.329482 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.329459    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c4087d-1e63-4c89-b69b-52cfd37db57b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9\" (UID: \"11c4087d-1e63-4c89-b69b-52cfd37db57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"
Oct 31 14:00:54.332279 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.332255    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c4087d-1e63-4c89-b69b-52cfd37db57b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9\" (UID: \"11c4087d-1e63-4c89-b69b-52cfd37db57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"
Oct 31 14:00:54.343261 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.343208    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsqld\" (UniqueName: \"kubernetes.io/projected/11c4087d-1e63-4c89-b69b-52cfd37db57b-kube-api-access-tsqld\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9\" (UID: \"11c4087d-1e63-4c89-b69b-52cfd37db57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"
Oct 31 14:00:54.487168 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.487132    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"]
Oct 31 14:00:54.492613 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.492590    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.496821 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.496796    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"
Oct 31 14:00:54.497344 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.497321    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-586469-openshift-default-dockercfg-kt8f6\""
Oct 31 14:00:54.505797 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.505771    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"]
Oct 31 14:00:54.532676 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.532624    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-workload-certs\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.532853 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.532809    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-data\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.532853 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.532841    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-token\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.533047 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.532898    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-credential-socket\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.533047 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.532929    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istiod-ca-cert\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.533047 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.532989    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-workload-socket\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.533296 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.533059    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-envoy\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.533296 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.533084    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-podinfo\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.533296 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.533213    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhxwg\" (UniqueName: \"kubernetes.io/projected/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-kube-api-access-dhxwg\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.580979 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.579713    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:54.635008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.634969    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-workload-socket\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.635201 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.635064    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-envoy\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.635201 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.635096    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-podinfo\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.635475 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.635214    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhxwg\" (UniqueName: \"kubernetes.io/projected/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-kube-api-access-dhxwg\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.635475 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.635292    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-workload-certs\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.635475 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.635441    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-data\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.635475 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.635467    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-token\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.635780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.635500    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-credential-socket\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.635780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.635527    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istiod-ca-cert\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.635918 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.635853    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-workload-socket\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.636188 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.636163    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-workload-certs\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.636315 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.636287    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istiod-ca-cert\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.636568 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.636545    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-credential-socket\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.637302 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.637273    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-data\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.643140 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.643111    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-podinfo\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.645089 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.645065    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-envoy\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.653845 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.653816    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-token\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.654753 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.654727    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhxwg\" (UniqueName: \"kubernetes.io/projected/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-kube-api-access-dhxwg\") pod \"e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") " pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.824003 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.821133    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:54.824003 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.823406    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:54.887910 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:54.875128    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"]
Oct 31 14:00:55.026300 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:55.026263    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"]
Oct 31 14:00:55.026570 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:55.026539    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5" podUID="4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" containerName="istio-proxy" containerID="cri-o://5fced279f9bcf70360c81f79e77f24bab3e322b6e96d6969d0c73bee00799e69" gracePeriod=30
Oct 31 14:00:55.068164 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:55.068132    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9" event={"ID":"11c4087d-1e63-4c89-b69b-52cfd37db57b","Type":"ContainerStarted","Data":"b899245c26f6ab04ac0639a1acf4471a310025d29af83a187695ce4af70c9a0c"}
Oct 31 14:00:55.243311 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:55.243277    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:55.307500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:55.307443    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"]
Oct 31 14:00:55.312578 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:00:55.312540    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5a5785_d16f_4b1f_ae6d_5f6c0fa50c21.slice/crio-a588ecbf8687c60fba4e1383ad0763ca8a3624c12a358d02ae50c985f8063ce4 WatchSource:0}: Error finding container a588ecbf8687c60fba4e1383ad0763ca8a3624c12a358d02ae50c985f8063ce4: Status 404 returned error can't find the container with id a588ecbf8687c60fba4e1383ad0763ca8a3624c12a358d02ae50c985f8063ce4
Oct 31 14:00:55.315906 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:55.315830    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:00:55.316067 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:55.316029    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:00:55.316150 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:55.316089    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:00:55.989069 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:55.989032    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:56.077549 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:56.077513    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m" event={"ID":"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21","Type":"ContainerStarted","Data":"d57653db8e69ce08b52d338fd62bdce1a79e7e92f409d88643ea1a6dbd738930"}
Oct 31 14:00:56.078074 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:56.077577    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m" event={"ID":"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21","Type":"ContainerStarted","Data":"a588ecbf8687c60fba4e1383ad0763ca8a3624c12a358d02ae50c985f8063ce4"}
Oct 31 14:00:56.079272 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:56.079247    4428 generic.go:356] "Generic (PLEG): container finished" podID="11c4087d-1e63-4c89-b69b-52cfd37db57b" containerID="0cedc121da47030aea7e014337fd8dcc61b3edde3a694addc9a956d5d94741f9" exitCode=0
Oct 31 14:00:56.079353 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:56.079335    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9" event={"ID":"11c4087d-1e63-4c89-b69b-52cfd37db57b","Type":"ContainerDied","Data":"0cedc121da47030aea7e014337fd8dcc61b3edde3a694addc9a956d5d94741f9"}
Oct 31 14:00:56.108092 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:56.107944    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m" podStartSLOduration=2.10792725 podStartE2EDuration="2.10792725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:56.103367241 +0000 UTC m=+3453.501715862" watchObservedRunningTime="2026-10-31 14:00:56.10792725 +0000 UTC m=+3453.506275873"
Oct 31 14:00:56.844081 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:56.844041    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:56.850107 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:56.850081    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:56.928889 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:56.925466    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:57.087606 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:57.087561    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:57.088700 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:57.088682    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:00:57.304459 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:57.304426    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"
Oct 31 14:00:57.456150 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:57.456004    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11c4087d-1e63-4c89-b69b-52cfd37db57b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c4087d-1e63-4c89-b69b-52cfd37db57b-maas-api-xks-ca\") pod \"11c4087d-1e63-4c89-b69b-52cfd37db57b\" (UID: \"11c4087d-1e63-4c89-b69b-52cfd37db57b\") "
Oct 31 14:00:57.456150 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:57.456052    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c4087d-1e63-4c89-b69b-52cfd37db57b-kube-api-access-tsqld\" (UniqueName: \"kubernetes.io/projected/11c4087d-1e63-4c89-b69b-52cfd37db57b-kube-api-access-tsqld\") pod \"11c4087d-1e63-4c89-b69b-52cfd37db57b\" (UID: \"11c4087d-1e63-4c89-b69b-52cfd37db57b\") "
Oct 31 14:00:57.456414 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:57.456226    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c4087d-1e63-4c89-b69b-52cfd37db57b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c4087d-1e63-4c89-b69b-52cfd37db57b-maas-api-openshift-ca\") pod \"11c4087d-1e63-4c89-b69b-52cfd37db57b\" (UID: \"11c4087d-1e63-4c89-b69b-52cfd37db57b\") "
Oct 31 14:00:57.456917 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:57.456885    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c4087d-1e63-4c89-b69b-52cfd37db57b-maas-api-openshift-ca" pod "11c4087d-1e63-4c89-b69b-52cfd37db57b" (UID: "11c4087d-1e63-4c89-b69b-52cfd37db57b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:00:57.457326 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:57.457299    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c4087d-1e63-4c89-b69b-52cfd37db57b-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:57.460518 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:57.460494    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c4087d-1e63-4c89-b69b-52cfd37db57b-maas-api-xks-ca" pod "11c4087d-1e63-4c89-b69b-52cfd37db57b" (UID: "11c4087d-1e63-4c89-b69b-52cfd37db57b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:00:57.460644 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:57.460555    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c4087d-1e63-4c89-b69b-52cfd37db57b-kube-api-access-tsqld" pod "11c4087d-1e63-4c89-b69b-52cfd37db57b" (UID: "11c4087d-1e63-4c89-b69b-52cfd37db57b"). InnerVolumeSpecName "kube-api-access-tsqld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:00:57.558684 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:57.558643    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c4087d-1e63-4c89-b69b-52cfd37db57b-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:57.558684 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:57.558676    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsqld\" (UniqueName: \"kubernetes.io/projected/11c4087d-1e63-4c89-b69b-52cfd37db57b-kube-api-access-tsqld\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:00:57.982893 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:57.980819    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:58.097233 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:58.097186    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9" event={"ID":"11c4087d-1e63-4c89-b69b-52cfd37db57b","Type":"ContainerDied","Data":"b899245c26f6ab04ac0639a1acf4471a310025d29af83a187695ce4af70c9a0c"}
Oct 31 14:00:58.097655 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:58.097247    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b899245c26f6ab04ac0639a1acf4471a310025d29af83a187695ce4af70c9a0c"
Oct 31 14:00:58.097655 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:58.097212    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"
Oct 31 14:00:58.580451 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:58.580414    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:58.746279 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:58.746246    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:59.157851 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:59.157817    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:59.646454 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:59.646407    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:59.859316 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:59.859279    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:00:59.871118 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:59.871079    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"]
Oct 31 14:00:59.871887 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:59.871854    4428 cpu_manager.go:414] "containerMap: removing container" podUID="11c4087d-1e63-4c89-b69b-52cfd37db57b" containerName="revoke-keys"
Oct 31 14:00:59.871887 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:59.871886    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c4087d-1e63-4c89-b69b-52cfd37db57b" containerName="revoke-keys"
Oct 31 14:00:59.872028 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:59.872022    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c4087d-1e63-4c89-b69b-52cfd37db57b" containerName="revoke-keys"
Oct 31 14:00:59.876044 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:59.876020    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"
Oct 31 14:00:59.886204 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:59.886176    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-586469-dockercfg-c9n7j\""
Oct 31 14:00:59.886580 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:59.886552    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-586469\""
Oct 31 14:00:59.888591 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:59.888565    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"]
Oct 31 14:00:59.924162 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:59.924068    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"]
Oct 31 14:00:59.928653 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:59.928632    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"
Oct 31 14:00:59.944835 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:00:59.944803    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"]
Oct 31 14:01:00.135028 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.106709    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6r7\" (UniqueName: \"kubernetes.io/projected/335e2046-6ac2-4332-a3d3-b721751681df-kube-api-access-fr6r7\") pod \"payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8\" (UID: \"335e2046-6ac2-4332-a3d3-b721751681df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"
Oct 31 14:01:00.135028 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.106798    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335e2046-6ac2-4332-a3d3-b721751681df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8\" (UID: \"335e2046-6ac2-4332-a3d3-b721751681df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"
Oct 31 14:01:00.135028 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.112165    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwcsj\" (UniqueName: \"kubernetes.io/projected/63f881f7-c955-411e-8c47-f87e0c94b581-kube-api-access-vwcsj\") pod \"payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc\" (UID: \"63f881f7-c955-411e-8c47-f87e0c94b581\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"
Oct 31 14:01:00.135028 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.120008    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:00.135028 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.121103    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f881f7-c955-411e-8c47-f87e0c94b581-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc\" (UID: \"63f881f7-c955-411e-8c47-f87e0c94b581\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"
Oct 31 14:01:00.281151 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.265661    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwcsj\" (UniqueName: \"kubernetes.io/projected/63f881f7-c955-411e-8c47-f87e0c94b581-kube-api-access-vwcsj\") pod \"payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc\" (UID: \"63f881f7-c955-411e-8c47-f87e0c94b581\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"
Oct 31 14:01:00.281151 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.265776    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f881f7-c955-411e-8c47-f87e0c94b581-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc\" (UID: \"63f881f7-c955-411e-8c47-f87e0c94b581\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"
Oct 31 14:01:00.281151 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.266014    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6r7\" (UniqueName: \"kubernetes.io/projected/335e2046-6ac2-4332-a3d3-b721751681df-kube-api-access-fr6r7\") pod \"payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8\" (UID: \"335e2046-6ac2-4332-a3d3-b721751681df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"
Oct 31 14:01:00.281151 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.266060    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335e2046-6ac2-4332-a3d3-b721751681df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8\" (UID: \"335e2046-6ac2-4332-a3d3-b721751681df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"
Oct 31 14:01:00.281151 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.273364    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f881f7-c955-411e-8c47-f87e0c94b581-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc\" (UID: \"63f881f7-c955-411e-8c47-f87e0c94b581\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"
Oct 31 14:01:00.281151 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.276662    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335e2046-6ac2-4332-a3d3-b721751681df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8\" (UID: \"335e2046-6ac2-4332-a3d3-b721751681df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"
Oct 31 14:01:00.283164 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.283133    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:00.287467 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.287441    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6r7\" (UniqueName: \"kubernetes.io/projected/335e2046-6ac2-4332-a3d3-b721751681df-kube-api-access-fr6r7\") pod \"payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8\" (UID: \"335e2046-6ac2-4332-a3d3-b721751681df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"
Oct 31 14:01:00.287467 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.287455    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwcsj\" (UniqueName: \"kubernetes.io/projected/63f881f7-c955-411e-8c47-f87e0c94b581-kube-api-access-vwcsj\") pod \"payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc\" (UID: \"63f881f7-c955-411e-8c47-f87e0c94b581\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"
Oct 31 14:01:00.476463 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.476422    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 14:01:00.493925 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.493846    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"
Oct 31 14:01:00.550112 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.550023    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"
Oct 31 14:01:00.571361 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.571311    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-credential-socket\") pod \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") "
Oct 31 14:01:00.571491 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.571386    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-envoy\") pod \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") "
Oct 31 14:01:00.571491 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.571425    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-workload-socket\") pod \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") "
Oct 31 14:01:00.571491 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.571467    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-data\") pod \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") "
Oct 31 14:01:00.571676 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.571515    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-token\" (UniqueName: \"kubernetes.io/projected/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-token\") pod \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") "
Oct 31 14:01:00.571676 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.571545    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-podinfo\") pod \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") "
Oct 31 14:01:00.571676 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.571599    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istiod-ca-cert\") pod \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") "
Oct 31 14:01:00.571676 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.571639    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-kube-api-access-9k2fb\" (UniqueName: \"kubernetes.io/projected/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-kube-api-access-9k2fb\") pod \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") "
Oct 31 14:01:00.571928 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.571696    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-workload-certs\") pod \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\" (UID: \"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e\") "
Oct 31 14:01:00.572139 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.572101    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-data" pod "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" (UID: "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:00.572263 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.572245    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.572615 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.572573    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-workload-certs" pod "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" (UID: "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:00.572717 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.572636    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-credential-socket" pod "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" (UID: "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:00.573122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.572908    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-workload-socket" pod "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" (UID: "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:00.573122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.573083    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istiod-ca-cert" pod "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" (UID: "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:00.575845 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.575811    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-envoy" pod "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" (UID: "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:00.576066 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.576034    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-kube-api-access-9k2fb" pod "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" (UID: "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e"). InnerVolumeSpecName "kube-api-access-9k2fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:00.577808 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.577781    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-token" pod "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" (UID: "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:00.578631 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.578589    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-podinfo" pod "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" (UID: "4e7c955c-ca35-4c0d-8625-b06d7b2bf73e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:01:00.674220 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.674183    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.674220 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.674217    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.674431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.674230    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k2fb\" (UniqueName: \"kubernetes.io/projected/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-kube-api-access-9k2fb\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.674431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.674244    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.674431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.674257    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.674431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.674269    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.674431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.674280    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.674431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.674291    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:00.709455 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.709420    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:00.799938 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.799745    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"]
Oct 31 14:01:00.800317 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:01:00.800264    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335e2046_6ac2_4332_a3d3_b721751681df.slice/crio-7e656d7486656519199f73bc1d787508bed0527f0665bf5e5ea4d09fc7816872 WatchSource:0}: Error finding container 7e656d7486656519199f73bc1d787508bed0527f0665bf5e5ea4d09fc7816872: Status 404 returned error can't find the container with id 7e656d7486656519199f73bc1d787508bed0527f0665bf5e5ea4d09fc7816872
Oct 31 14:01:00.865200 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:00.865171    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"]
Oct 31 14:01:00.866053 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:01:00.866024    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f881f7_c955_411e_8c47_f87e0c94b581.slice/crio-434a32c72757078af05190f975bea569a2d919d6ff5ca2f63cb951e9bf06b023 WatchSource:0}: Error finding container 434a32c72757078af05190f975bea569a2d919d6ff5ca2f63cb951e9bf06b023: Status 404 returned error can't find the container with id 434a32c72757078af05190f975bea569a2d919d6ff5ca2f63cb951e9bf06b023
Oct 31 14:01:01.024953 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.019406    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:01.207113 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.207068    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8" event={"ID":"335e2046-6ac2-4332-a3d3-b721751681df","Type":"ContainerStarted","Data":"72c31c199f1ced1b81b30de9b02871d16bd06f3cf4225d41efd2d26897a3a776"}
Oct 31 14:01:01.207289 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.207146    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8" event={"ID":"335e2046-6ac2-4332-a3d3-b721751681df","Type":"ContainerStarted","Data":"7e656d7486656519199f73bc1d787508bed0527f0665bf5e5ea4d09fc7816872"}
Oct 31 14:01:01.207289 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.207264    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"
Oct 31 14:01:01.208969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.208905    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc" event={"ID":"63f881f7-c955-411e-8c47-f87e0c94b581","Type":"ContainerStarted","Data":"90e8601b0f8153fdc7322c0f79b91ee8c65bb789d22833a17e893d2e0bbd58bd"}
Oct 31 14:01:01.208969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.208963    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc" event={"ID":"63f881f7-c955-411e-8c47-f87e0c94b581","Type":"ContainerStarted","Data":"434a32c72757078af05190f975bea569a2d919d6ff5ca2f63cb951e9bf06b023"}
Oct 31 14:01:01.209159 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.208994    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"
Oct 31 14:01:01.210761 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.210734    4428 generic.go:356] "Generic (PLEG): container finished" podID="4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" containerID="5fced279f9bcf70360c81f79e77f24bab3e322b6e96d6969d0c73bee00799e69" exitCode=0
Oct 31 14:01:01.210921 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.210781    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"
Oct 31 14:01:01.210921 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.210780    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5" event={"ID":"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e","Type":"ContainerDied","Data":"5fced279f9bcf70360c81f79e77f24bab3e322b6e96d6969d0c73bee00799e69"}
Oct 31 14:01:01.211044 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.210936    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5" event={"ID":"4e7c955c-ca35-4c0d-8625-b06d7b2bf73e","Type":"ContainerDied","Data":"6810a13671a376dcc5a6a333980ee58dcee333da8aa8dea41e4c5ad8d7a9d0e0"}
Oct 31 14:01:01.211044 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.210976    4428 scope.go:122] "RemoveContainer" containerID="5fced279f9bcf70360c81f79e77f24bab3e322b6e96d6969d0c73bee00799e69"
Oct 31 14:01:01.224285 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.224261    4428 scope.go:122] "RemoveContainer" containerID="694883a8256ff938e1e8e8358a22a3d5b7b1ed9c6480a4188447472456c64804"
Oct 31 14:01:01.230908 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.228572    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8" podStartSLOduration=2.228555467 podStartE2EDuration="2.228555467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:01:01.228192546 +0000 UTC m=+3458.626541170" watchObservedRunningTime="2026-10-31 14:01:01.228555467 +0000 UTC m=+3458.626904090"
Oct 31 14:01:01.240721 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.240694    4428 scope.go:122] "RemoveContainer" containerID="5fced279f9bcf70360c81f79e77f24bab3e322b6e96d6969d0c73bee00799e69"
Oct 31 14:01:01.241092 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:01:01.241064    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fced279f9bcf70360c81f79e77f24bab3e322b6e96d6969d0c73bee00799e69\": container with ID starting with 5fced279f9bcf70360c81f79e77f24bab3e322b6e96d6969d0c73bee00799e69 not found: ID does not exist" containerID="5fced279f9bcf70360c81f79e77f24bab3e322b6e96d6969d0c73bee00799e69"
Oct 31 14:01:01.241227 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.241099    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fced279f9bcf70360c81f79e77f24bab3e322b6e96d6969d0c73bee00799e69"} err="failed to get container status \"5fced279f9bcf70360c81f79e77f24bab3e322b6e96d6969d0c73bee00799e69\": rpc error: code = NotFound desc = could not find container \"5fced279f9bcf70360c81f79e77f24bab3e322b6e96d6969d0c73bee00799e69\": container with ID starting with 5fced279f9bcf70360c81f79e77f24bab3e322b6e96d6969d0c73bee00799e69 not found: ID does not exist"
Oct 31 14:01:01.241227 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.241118    4428 scope.go:122] "RemoveContainer" containerID="694883a8256ff938e1e8e8358a22a3d5b7b1ed9c6480a4188447472456c64804"
Oct 31 14:01:01.241455 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:01:01.241436    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"694883a8256ff938e1e8e8358a22a3d5b7b1ed9c6480a4188447472456c64804\": container with ID starting with 694883a8256ff938e1e8e8358a22a3d5b7b1ed9c6480a4188447472456c64804 not found: ID does not exist" containerID="694883a8256ff938e1e8e8358a22a3d5b7b1ed9c6480a4188447472456c64804"
Oct 31 14:01:01.241501 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.241463    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"694883a8256ff938e1e8e8358a22a3d5b7b1ed9c6480a4188447472456c64804"} err="failed to get container status \"694883a8256ff938e1e8e8358a22a3d5b7b1ed9c6480a4188447472456c64804\": rpc error: code = NotFound desc = could not find container \"694883a8256ff938e1e8e8358a22a3d5b7b1ed9c6480a4188447472456c64804\": container with ID starting with 694883a8256ff938e1e8e8358a22a3d5b7b1ed9c6480a4188447472456c64804 not found: ID does not exist"
Oct 31 14:01:01.256907 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.256822    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc" podStartSLOduration=2.256803546 podStartE2EDuration="2.256803546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:01:01.25384254 +0000 UTC m=+3458.652191166" watchObservedRunningTime="2026-10-31 14:01:01.256803546 +0000 UTC m=+3458.655152174"
Oct 31 14:01:01.272311 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.272274    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"]
Oct 31 14:01:01.277002 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.276975    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-qbxs5"]
Oct 31 14:01:01.969895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:01.966433    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:02.756789 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:02.756752    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" path="/var/lib/kubelet/pods/4e7c955c-ca35-4c0d-8625-b06d7b2bf73e/volumes"
Oct 31 14:01:02.987443 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:02.987284    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:03.812568 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:03.812530    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:03.984062 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:03.983981    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:04.868365 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:04.868320    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:05.703677 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:05.703630    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:05.783844 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:05.783807    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"]
Oct 31 14:01:05.919325 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:05.919294    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:06.092898 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:06.089743    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:06.544910 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:06.537854    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:07.452521 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:07.449970    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:07.766144 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:07.764815    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:07.923328 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:07.923293    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:07.958073 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:07.958029    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"]
Oct 31 14:01:07.989077 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:07.989036    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"]
Oct 31 14:01:07.989676 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:07.989626    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4" podUID="54b8f412-0415-4f74-9f20-47d392f1cb00" containerName="istio-proxy" containerID="cri-o://3e232e7c304418539d851d01d4eb7d30f647dd7803a6ee75b61c7ba89a24ff49" gracePeriod=30
Oct 31 14:01:08.060611 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.057970    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"]
Oct 31 14:01:08.060611 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.058287    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk" podUID="4c0d42c0-825f-4d1f-9670-e695dfe168b1" containerName="istio-proxy" containerID="cri-o://c278b0991f0b458376a8e14a9790afa4f412e40ba983eac01fa1a78b83c7edcb" gracePeriod=30
Oct 31 14:01:08.070726 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.069564    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"]
Oct 31 14:01:08.070726 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.069881    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p" podUID="91b3d413-e7cb-4505-96b3-d7da1f9e0d85" containerName="istio-proxy" containerID="cri-o://30eceb9873c2cd909fb1fd0a7761ae837f81d0e21f7559ff2f2e77b81f0977e5" gracePeriod=30
Oct 31 14:01:08.108337 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.108302    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:08.428123 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.428015    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"]
Oct 31 14:01:08.428828 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.428809    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" containerName="istio-proxy"
Oct 31 14:01:08.428828 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.428828    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" containerName="istio-proxy"
Oct 31 14:01:08.429028 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.428845    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" containerName="istio-proxy"
Oct 31 14:01:08.429028 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.428852    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" containerName="istio-proxy"
Oct 31 14:01:08.429028 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.428985    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" containerName="istio-proxy"
Oct 31 14:01:08.429028 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.428996    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7c955c-ca35-4c0d-8625-b06d7b2bf73e" containerName="istio-proxy"
Oct 31 14:01:08.437056 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.437031    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"
Oct 31 14:01:08.445898 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.442169    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:01:08.445898 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.444327    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:01:08.449891 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.448548    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"]
Oct 31 14:01:08.594984 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.594918    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570a38b1-3f0e-4a64-a66b-24786d0813a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l\" (UID: \"570a38b1-3f0e-4a64-a66b-24786d0813a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"
Oct 31 14:01:08.595688 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.595659    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570a38b1-3f0e-4a64-a66b-24786d0813a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l\" (UID: \"570a38b1-3f0e-4a64-a66b-24786d0813a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"
Oct 31 14:01:08.595807 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.595794    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxs5d\" (UniqueName: \"kubernetes.io/projected/570a38b1-3f0e-4a64-a66b-24786d0813a0-kube-api-access-xxs5d\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l\" (UID: \"570a38b1-3f0e-4a64-a66b-24786d0813a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"
Oct 31 14:01:08.616563 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.616534    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:08.697294 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.697185    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxs5d\" (UniqueName: \"kubernetes.io/projected/570a38b1-3f0e-4a64-a66b-24786d0813a0-kube-api-access-xxs5d\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l\" (UID: \"570a38b1-3f0e-4a64-a66b-24786d0813a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"
Oct 31 14:01:08.697509 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.697333    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570a38b1-3f0e-4a64-a66b-24786d0813a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l\" (UID: \"570a38b1-3f0e-4a64-a66b-24786d0813a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"
Oct 31 14:01:08.697509 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.697485    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570a38b1-3f0e-4a64-a66b-24786d0813a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l\" (UID: \"570a38b1-3f0e-4a64-a66b-24786d0813a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"
Oct 31 14:01:08.698407 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.698378    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570a38b1-3f0e-4a64-a66b-24786d0813a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l\" (UID: \"570a38b1-3f0e-4a64-a66b-24786d0813a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"
Oct 31 14:01:08.700003 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.699981    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570a38b1-3f0e-4a64-a66b-24786d0813a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l\" (UID: \"570a38b1-3f0e-4a64-a66b-24786d0813a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"
Oct 31 14:01:08.713799 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.713775    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxs5d\" (UniqueName: \"kubernetes.io/projected/570a38b1-3f0e-4a64-a66b-24786d0813a0-kube-api-access-xxs5d\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l\" (UID: \"570a38b1-3f0e-4a64-a66b-24786d0813a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"
Oct 31 14:01:08.760439 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:08.760395    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"
Oct 31 14:01:09.066282 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:09.066248    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"]
Oct 31 14:01:09.067555 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:01:09.067527    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570a38b1_3f0e_4a64_a66b_24786d0813a0.slice/crio-bf0000c9c9a9487888e0fa6f52d38f86b65544a48f15040f0041ae9de56559b1 WatchSource:0}: Error finding container bf0000c9c9a9487888e0fa6f52d38f86b65544a48f15040f0041ae9de56559b1: Status 404 returned error can't find the container with id bf0000c9c9a9487888e0fa6f52d38f86b65544a48f15040f0041ae9de56559b1
Oct 31 14:01:09.293679 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:09.293641    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l" event={"ID":"570a38b1-3f0e-4a64-a66b-24786d0813a0","Type":"ContainerStarted","Data":"bf0000c9c9a9487888e0fa6f52d38f86b65544a48f15040f0041ae9de56559b1"}
Oct 31 14:01:09.705217 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:09.705059    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:10.094228 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:10.094191    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:01:10.094368 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:10.094258    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p" podUID="91b3d413-e7cb-4505-96b3-d7da1f9e0d85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:01:10.247346 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:10.247300    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:01:10.247517 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:10.247356    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4" podUID="54b8f412-0415-4f74-9f20-47d392f1cb00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:01:10.302414 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:10.302375    4428 generic.go:356] "Generic (PLEG): container finished" podID="570a38b1-3f0e-4a64-a66b-24786d0813a0" containerID="a740dc3d9beabaea02feb2ce7f21dc4c90e229b4fedde717fdd8d21d49aa7fa0" exitCode=0
Oct 31 14:01:10.302606 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:10.302467    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l" event={"ID":"570a38b1-3f0e-4a64-a66b-24786d0813a0","Type":"ContainerDied","Data":"a740dc3d9beabaea02feb2ce7f21dc4c90e229b4fedde717fdd8d21d49aa7fa0"}
Oct 31 14:01:10.755461 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:10.755427    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:10.784509 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:10.784470    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv" podUID="71b06a52-1d53-48e7-a5bd-b25f92a89e91" containerName="payload-processing" containerID="cri-o://8b699afc29f2d8d9241147d585c8407538f20577735eb3be8321b037c9fc1c3c" gracePeriod=25
Oct 31 14:01:11.477307 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:11.477276    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"
Oct 31 14:01:11.575488 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:11.575340    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570a38b1-3f0e-4a64-a66b-24786d0813a0-kube-api-access-xxs5d\" (UniqueName: \"kubernetes.io/projected/570a38b1-3f0e-4a64-a66b-24786d0813a0-kube-api-access-xxs5d\") pod \"570a38b1-3f0e-4a64-a66b-24786d0813a0\" (UID: \"570a38b1-3f0e-4a64-a66b-24786d0813a0\") "
Oct 31 14:01:11.600915 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:11.584618    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/570a38b1-3f0e-4a64-a66b-24786d0813a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570a38b1-3f0e-4a64-a66b-24786d0813a0-maas-api-xks-ca\") pod \"570a38b1-3f0e-4a64-a66b-24786d0813a0\" (UID: \"570a38b1-3f0e-4a64-a66b-24786d0813a0\") "
Oct 31 14:01:11.600915 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:11.598858    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570a38b1-3f0e-4a64-a66b-24786d0813a0-kube-api-access-xxs5d" pod "570a38b1-3f0e-4a64-a66b-24786d0813a0" (UID: "570a38b1-3f0e-4a64-a66b-24786d0813a0"). InnerVolumeSpecName "kube-api-access-xxs5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:11.600915 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:11.600772    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/570a38b1-3f0e-4a64-a66b-24786d0813a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570a38b1-3f0e-4a64-a66b-24786d0813a0-maas-api-openshift-ca\") pod \"570a38b1-3f0e-4a64-a66b-24786d0813a0\" (UID: \"570a38b1-3f0e-4a64-a66b-24786d0813a0\") "
Oct 31 14:01:11.602472 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:11.601707    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxs5d\" (UniqueName: \"kubernetes.io/projected/570a38b1-3f0e-4a64-a66b-24786d0813a0-kube-api-access-xxs5d\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:11.602472 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:11.602094    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570a38b1-3f0e-4a64-a66b-24786d0813a0-maas-api-openshift-ca" pod "570a38b1-3f0e-4a64-a66b-24786d0813a0" (UID: "570a38b1-3f0e-4a64-a66b-24786d0813a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:11.604054 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:11.604020    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/570a38b1-3f0e-4a64-a66b-24786d0813a0-maas-api-xks-ca" pod "570a38b1-3f0e-4a64-a66b-24786d0813a0" (UID: "570a38b1-3f0e-4a64-a66b-24786d0813a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:11.604838 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:11.604690    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:11.702912 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:11.702840    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570a38b1-3f0e-4a64-a66b-24786d0813a0-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:11.702912 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:11.702889    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570a38b1-3f0e-4a64-a66b-24786d0813a0-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:12.223638 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:12.223606    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"
Oct 31 14:01:12.224183 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:12.223770    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"
Oct 31 14:01:12.249925 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:12.249850    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:01:12.250146 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:12.249953    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk" podUID="4c0d42c0-825f-4d1f-9670-e695dfe168b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:01:12.323672 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:12.323633    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"
Oct 31 14:01:12.323672 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:12.323657    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l" event={"ID":"570a38b1-3f0e-4a64-a66b-24786d0813a0","Type":"ContainerDied","Data":"bf0000c9c9a9487888e0fa6f52d38f86b65544a48f15040f0041ae9de56559b1"}
Oct 31 14:01:12.323949 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:12.323755    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf0000c9c9a9487888e0fa6f52d38f86b65544a48f15040f0041ae9de56559b1"
Oct 31 14:01:12.416333 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:12.416194    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:13.313305 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.310530    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:13.366938 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.363708    4428 generic.go:356] "Generic (PLEG): container finished" podID="91b3d413-e7cb-4505-96b3-d7da1f9e0d85" containerID="30eceb9873c2cd909fb1fd0a7761ae837f81d0e21f7559ff2f2e77b81f0977e5" exitCode=0
Oct 31 14:01:13.366938 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.363831    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p" event={"ID":"91b3d413-e7cb-4505-96b3-d7da1f9e0d85","Type":"ContainerDied","Data":"30eceb9873c2cd909fb1fd0a7761ae837f81d0e21f7559ff2f2e77b81f0977e5"}
Oct 31 14:01:13.366938 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.366339    4428 generic.go:356] "Generic (PLEG): container finished" podID="54b8f412-0415-4f74-9f20-47d392f1cb00" containerID="3e232e7c304418539d851d01d4eb7d30f647dd7803a6ee75b61c7ba89a24ff49" exitCode=0
Oct 31 14:01:13.366938 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.366485    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4" event={"ID":"54b8f412-0415-4f74-9f20-47d392f1cb00","Type":"ContainerDied","Data":"3e232e7c304418539d851d01d4eb7d30f647dd7803a6ee75b61c7ba89a24ff49"}
Oct 31 14:01:13.369653 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.369257    4428 generic.go:356] "Generic (PLEG): container finished" podID="4c0d42c0-825f-4d1f-9670-e695dfe168b1" containerID="c278b0991f0b458376a8e14a9790afa4f412e40ba983eac01fa1a78b83c7edcb" exitCode=0
Oct 31 14:01:13.369653 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.369387    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk" event={"ID":"4c0d42c0-825f-4d1f-9670-e695dfe168b1","Type":"ContainerDied","Data":"c278b0991f0b458376a8e14a9790afa4f412e40ba983eac01fa1a78b83c7edcb"}
Oct 31 14:01:13.449413 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.449378    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 14:01:13.458476 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.458445    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 14:01:13.545668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.539498    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-podinfo\") pod \"54b8f412-0415-4f74-9f20-47d392f1cb00\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") "
Oct 31 14:01:13.545668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.539611    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-envoy\") pod \"54b8f412-0415-4f74-9f20-47d392f1cb00\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") "
Oct 31 14:01:13.545668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.539653    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-workload-socket\") pod \"54b8f412-0415-4f74-9f20-47d392f1cb00\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") "
Oct 31 14:01:13.545668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.539748    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-token\" (UniqueName: \"kubernetes.io/projected/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-token\") pod \"54b8f412-0415-4f74-9f20-47d392f1cb00\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") "
Oct 31 14:01:13.545668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.539826    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b8f412-0415-4f74-9f20-47d392f1cb00-kube-api-access-89c6w\" (UniqueName: \"kubernetes.io/projected/54b8f412-0415-4f74-9f20-47d392f1cb00-kube-api-access-89c6w\") pod \"54b8f412-0415-4f74-9f20-47d392f1cb00\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") "
Oct 31 14:01:13.545668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.540719    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-credential-socket\") pod \"54b8f412-0415-4f74-9f20-47d392f1cb00\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") "
Oct 31 14:01:13.545668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.540797    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-workload-certs\") pod \"54b8f412-0415-4f74-9f20-47d392f1cb00\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") "
Oct 31 14:01:13.545668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.540825    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-data\") pod \"54b8f412-0415-4f74-9f20-47d392f1cb00\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") "
Oct 31 14:01:13.545668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.540842    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b8f412-0415-4f74-9f20-47d392f1cb00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b8f412-0415-4f74-9f20-47d392f1cb00-istiod-ca-cert\") pod \"54b8f412-0415-4f74-9f20-47d392f1cb00\" (UID: \"54b8f412-0415-4f74-9f20-47d392f1cb00\") "
Oct 31 14:01:13.545668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.542061    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 14:01:13.545668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.542082    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b8f412-0415-4f74-9f20-47d392f1cb00-istiod-ca-cert" pod "54b8f412-0415-4f74-9f20-47d392f1cb00" (UID: "54b8f412-0415-4f74-9f20-47d392f1cb00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:13.545668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.542800    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-workload-certs" pod "54b8f412-0415-4f74-9f20-47d392f1cb00" (UID: "54b8f412-0415-4f74-9f20-47d392f1cb00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.545668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.543040    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-credential-socket" pod "54b8f412-0415-4f74-9f20-47d392f1cb00" (UID: "54b8f412-0415-4f74-9f20-47d392f1cb00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.545668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.543218    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-data" pod "54b8f412-0415-4f74-9f20-47d392f1cb00" (UID: "54b8f412-0415-4f74-9f20-47d392f1cb00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.545668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.543423    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-workload-socket" pod "54b8f412-0415-4f74-9f20-47d392f1cb00" (UID: "54b8f412-0415-4f74-9f20-47d392f1cb00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.545668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.545069    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-token" pod "54b8f412-0415-4f74-9f20-47d392f1cb00" (UID: "54b8f412-0415-4f74-9f20-47d392f1cb00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:13.546774 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.546517    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-envoy" pod "54b8f412-0415-4f74-9f20-47d392f1cb00" (UID: "54b8f412-0415-4f74-9f20-47d392f1cb00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.547250 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.547224    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b8f412-0415-4f74-9f20-47d392f1cb00-kube-api-access-89c6w" pod "54b8f412-0415-4f74-9f20-47d392f1cb00" (UID: "54b8f412-0415-4f74-9f20-47d392f1cb00"). InnerVolumeSpecName "kube-api-access-89c6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:13.548091 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.548046    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-podinfo" pod "54b8f412-0415-4f74-9f20-47d392f1cb00" (UID: "54b8f412-0415-4f74-9f20-47d392f1cb00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:01:13.642694 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.642603    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-podinfo\") pod \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") "
Oct 31 14:01:13.642694 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.642645    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-token\" (UniqueName: \"kubernetes.io/projected/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-token\") pod \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") "
Oct 31 14:01:13.642694 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.642672    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istiod-ca-cert\") pod \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") "
Oct 31 14:01:13.643026 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.642701    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-data\") pod \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") "
Oct 31 14:01:13.643026 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.642726    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-workload-certs\") pod \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") "
Oct 31 14:01:13.643026 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.642742    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-data\") pod \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") "
Oct 31 14:01:13.643026 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.642821    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-kube-api-access-x6k74\" (UniqueName: \"kubernetes.io/projected/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-kube-api-access-x6k74\") pod \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") "
Oct 31 14:01:13.643026 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.642853    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-envoy\") pod \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") "
Oct 31 14:01:13.643026 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.642938    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-credential-socket\") pod \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") "
Oct 31 14:01:13.643026 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.642979    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-credential-socket\") pod \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") "
Oct 31 14:01:13.643394 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.643043    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istiod-ca-cert\") pod \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") "
Oct 31 14:01:13.643394 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.643042    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-data" pod "4c0d42c0-825f-4d1f-9670-e695dfe168b1" (UID: "4c0d42c0-825f-4d1f-9670-e695dfe168b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.643394 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.643214    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istiod-ca-cert" pod "91b3d413-e7cb-4505-96b3-d7da1f9e0d85" (UID: "91b3d413-e7cb-4505-96b3-d7da1f9e0d85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:13.643394 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.643332    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-workload-socket" pod "4c0d42c0-825f-4d1f-9670-e695dfe168b1" (UID: "4c0d42c0-825f-4d1f-9670-e695dfe168b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.643394 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.643085    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-workload-socket\") pod \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") "
Oct 31 14:01:13.643662 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.643459    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c0d42c0-825f-4d1f-9670-e695dfe168b1-kube-api-access-2gwx2\" (UniqueName: \"kubernetes.io/projected/4c0d42c0-825f-4d1f-9670-e695dfe168b1-kube-api-access-2gwx2\") pod \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") "
Oct 31 14:01:13.643662 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.643501    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-token\" (UniqueName: \"kubernetes.io/projected/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-token\") pod \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") "
Oct 31 14:01:13.643662 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.643504    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-credential-socket" pod "4c0d42c0-825f-4d1f-9670-e695dfe168b1" (UID: "4c0d42c0-825f-4d1f-9670-e695dfe168b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.643662 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.643558    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-credential-socket" pod "91b3d413-e7cb-4505-96b3-d7da1f9e0d85" (UID: "91b3d413-e7cb-4505-96b3-d7da1f9e0d85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.643662 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.643583    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-envoy\") pod \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") "
Oct 31 14:01:13.643662 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.643660    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-workload-certs\") pod \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\" (UID: \"4c0d42c0-825f-4d1f-9670-e695dfe168b1\") "
Oct 31 14:01:13.644047 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.643832    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-podinfo\") pod \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") "
Oct 31 14:01:13.644047 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.643851    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istiod-ca-cert" pod "4c0d42c0-825f-4d1f-9670-e695dfe168b1" (UID: "4c0d42c0-825f-4d1f-9670-e695dfe168b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:13.644047 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.643902    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-workload-socket\") pod \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\" (UID: \"91b3d413-e7cb-4505-96b3-d7da1f9e0d85\") "
Oct 31 14:01:13.644047 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.644027    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-workload-certs" pod "91b3d413-e7cb-4505-96b3-d7da1f9e0d85" (UID: "91b3d413-e7cb-4505-96b3-d7da1f9e0d85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.644420 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.644397    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-data" pod "91b3d413-e7cb-4505-96b3-d7da1f9e0d85" (UID: "91b3d413-e7cb-4505-96b3-d7da1f9e0d85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.644675 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.644645    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-workload-socket" pod "91b3d413-e7cb-4505-96b3-d7da1f9e0d85" (UID: "91b3d413-e7cb-4505-96b3-d7da1f9e0d85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.644852 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.644831    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-workload-certs" pod "4c0d42c0-825f-4d1f-9670-e695dfe168b1" (UID: "4c0d42c0-825f-4d1f-9670-e695dfe168b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.645084 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645060    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645151 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645091    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645151 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645108    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b8f412-0415-4f74-9f20-47d392f1cb00-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645151 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645126    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645151 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645142    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645358 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645157    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645358 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645175    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645358 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645190    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645358 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645204    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645358 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645218    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645358 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645232    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645358 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645246    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b8f412-0415-4f74-9f20-47d392f1cb00-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645358 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645260    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645358 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645274    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645358 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645290    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645358 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645304    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89c6w\" (UniqueName: \"kubernetes.io/projected/54b8f412-0415-4f74-9f20-47d392f1cb00-kube-api-access-89c6w\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645358 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645322    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645358 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645337    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.645358 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645349    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b8f412-0415-4f74-9f20-47d392f1cb00-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.646165 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.645924    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-token" pod "91b3d413-e7cb-4505-96b3-d7da1f9e0d85" (UID: "91b3d413-e7cb-4505-96b3-d7da1f9e0d85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:13.646265 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.646244    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-envoy" pod "91b3d413-e7cb-4505-96b3-d7da1f9e0d85" (UID: "91b3d413-e7cb-4505-96b3-d7da1f9e0d85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.646606 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.646575    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-podinfo" pod "4c0d42c0-825f-4d1f-9670-e695dfe168b1" (UID: "4c0d42c0-825f-4d1f-9670-e695dfe168b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:01:13.646850 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.646824    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-envoy" pod "4c0d42c0-825f-4d1f-9670-e695dfe168b1" (UID: "4c0d42c0-825f-4d1f-9670-e695dfe168b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:13.647247 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.647223    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0d42c0-825f-4d1f-9670-e695dfe168b1-kube-api-access-2gwx2" pod "4c0d42c0-825f-4d1f-9670-e695dfe168b1" (UID: "4c0d42c0-825f-4d1f-9670-e695dfe168b1"). InnerVolumeSpecName "kube-api-access-2gwx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:13.647445 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.647416    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-kube-api-access-x6k74" pod "91b3d413-e7cb-4505-96b3-d7da1f9e0d85" (UID: "91b3d413-e7cb-4505-96b3-d7da1f9e0d85"). InnerVolumeSpecName "kube-api-access-x6k74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:13.647704 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.647681    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-token" pod "4c0d42c0-825f-4d1f-9670-e695dfe168b1" (UID: "4c0d42c0-825f-4d1f-9670-e695dfe168b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:13.648213 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.648195    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-podinfo" pod "91b3d413-e7cb-4505-96b3-d7da1f9e0d85" (UID: "91b3d413-e7cb-4505-96b3-d7da1f9e0d85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:01:13.747002 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.746969    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6k74\" (UniqueName: \"kubernetes.io/projected/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-kube-api-access-x6k74\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.747002 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.746999    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.747002 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.747010    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gwx2\" (UniqueName: \"kubernetes.io/projected/4c0d42c0-825f-4d1f-9670-e695dfe168b1-kube-api-access-2gwx2\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.747267 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.747018    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.747267 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.747028    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.747267 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.747038    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.747267 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.747047    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0d42c0-825f-4d1f-9670-e695dfe168b1-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:13.747267 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:13.747055    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b3d413-e7cb-4505-96b3-d7da1f9e0d85-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:14.176169 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.176134    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"]
Oct 31 14:01:14.176924 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.176898    4428 cpu_manager.go:414] "containerMap: removing container" podUID="54b8f412-0415-4f74-9f20-47d392f1cb00" containerName="istio-proxy"
Oct 31 14:01:14.176924 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.176919    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b8f412-0415-4f74-9f20-47d392f1cb00" containerName="istio-proxy"
Oct 31 14:01:14.177122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.176935    4428 cpu_manager.go:414] "containerMap: removing container" podUID="91b3d413-e7cb-4505-96b3-d7da1f9e0d85" containerName="istio-proxy"
Oct 31 14:01:14.177122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.176941    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b3d413-e7cb-4505-96b3-d7da1f9e0d85" containerName="istio-proxy"
Oct 31 14:01:14.177122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.176973    4428 cpu_manager.go:414] "containerMap: removing container" podUID="570a38b1-3f0e-4a64-a66b-24786d0813a0" containerName="revoke-keys"
Oct 31 14:01:14.177122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.176978    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570a38b1-3f0e-4a64-a66b-24786d0813a0" containerName="revoke-keys"
Oct 31 14:01:14.177122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.176995    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4c0d42c0-825f-4d1f-9670-e695dfe168b1" containerName="istio-proxy"
Oct 31 14:01:14.177122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.177001    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c0d42c0-825f-4d1f-9670-e695dfe168b1" containerName="istio-proxy"
Oct 31 14:01:14.177122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.177105    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b8f412-0415-4f74-9f20-47d392f1cb00" containerName="istio-proxy"
Oct 31 14:01:14.177122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.177115    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b3d413-e7cb-4505-96b3-d7da1f9e0d85" containerName="istio-proxy"
Oct 31 14:01:14.177122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.177126    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="570a38b1-3f0e-4a64-a66b-24786d0813a0" containerName="revoke-keys"
Oct 31 14:01:14.177492 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.177134    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c0d42c0-825f-4d1f-9670-e695dfe168b1" containerName="istio-proxy"
Oct 31 14:01:14.182677 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.182653    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:01:14.193524 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.193496    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-586469-serving-cert\""
Oct 31 14:01:14.195056 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.195025    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"]
Oct 31 14:01:14.343300 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.342704    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:14.356077 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.356036    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl\" (UID: \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:01:14.356302 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.356088    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rztr\" (UniqueName: \"kubernetes.io/projected/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-kube-api-access-7rztr\") pod \"maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl\" (UID: \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:01:14.356392 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.356319    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-maas-api-tls\") pod \"maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl\" (UID: \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:01:14.356392 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.356363    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-metrics-certs\") pod \"maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl\" (UID: \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:01:14.378946 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.378915    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"
Oct 31 14:01:14.379167 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.378905    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p" event={"ID":"91b3d413-e7cb-4505-96b3-d7da1f9e0d85","Type":"ContainerDied","Data":"1abf6ccd8e76f82b518aaf3d9de337366936323ec385490aacf3b723468b0a4f"}
Oct 31 14:01:14.379167 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.379088    4428 scope.go:122] "RemoveContainer" containerID="30eceb9873c2cd909fb1fd0a7761ae837f81d0e21f7559ff2f2e77b81f0977e5"
Oct 31 14:01:14.380598 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.380541    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4" event={"ID":"54b8f412-0415-4f74-9f20-47d392f1cb00","Type":"ContainerDied","Data":"224de854e0149c1d48be58d874d1436b72c1ff8d1acfd17f1cef3a18edbdcbad"}
Oct 31 14:01:14.380598 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.380558    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"
Oct 31 14:01:14.382333 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.382313    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"
Oct 31 14:01:14.382469 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.382354    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk" event={"ID":"4c0d42c0-825f-4d1f-9670-e695dfe168b1","Type":"ContainerDied","Data":"1be225a9a309736f38834ef08924474b694d0a165b847422fee410e9e390311d"}
Oct 31 14:01:14.393836 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.393800    4428 scope.go:122] "RemoveContainer" containerID="3e232e7c304418539d851d01d4eb7d30f647dd7803a6ee75b61c7ba89a24ff49"
Oct 31 14:01:14.405994 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.405973    4428 scope.go:122] "RemoveContainer" containerID="c278b0991f0b458376a8e14a9790afa4f412e40ba983eac01fa1a78b83c7edcb"
Oct 31 14:01:14.415389 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.415317    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"]
Oct 31 14:01:14.426827 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.424116    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-g2x4p"]
Oct 31 14:01:14.434526 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.434491    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"]
Oct 31 14:01:14.438930 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.438902    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-p5gk4"]
Oct 31 14:01:14.453008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.452979    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"]
Oct 31 14:01:14.457685 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.457646    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-maas-api-tls\") pod \"maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl\" (UID: \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:01:14.457858 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.457701    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-metrics-certs\") pod \"maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl\" (UID: \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:01:14.457858 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.457808    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl\" (UID: \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:01:14.457858 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.457834    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rztr\" (UniqueName: \"kubernetes.io/projected/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-kube-api-access-7rztr\") pod \"maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl\" (UID: \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:01:14.458567 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.458532    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl\" (UID: \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:01:14.460433 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.460402    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-maas-api-tls\") pod \"maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl\" (UID: \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:01:14.460859 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.460837    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-wg5vk"]
Oct 31 14:01:14.460960 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.460938    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-metrics-certs\") pod \"maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl\" (UID: \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:01:14.470992 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.470968    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rztr\" (UniqueName: \"kubernetes.io/projected/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-kube-api-access-7rztr\") pod \"maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl\" (UID: \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:01:14.497804 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.497763    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:14.514072 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.514042    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:01:14.757094 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.757058    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c0d42c0-825f-4d1f-9670-e695dfe168b1" path="/var/lib/kubelet/pods/4c0d42c0-825f-4d1f-9670-e695dfe168b1/volumes"
Oct 31 14:01:14.757630 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.757615    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b8f412-0415-4f74-9f20-47d392f1cb00" path="/var/lib/kubelet/pods/54b8f412-0415-4f74-9f20-47d392f1cb00/volumes"
Oct 31 14:01:14.758070 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.758056    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b3d413-e7cb-4505-96b3-d7da1f9e0d85" path="/var/lib/kubelet/pods/91b3d413-e7cb-4505-96b3-d7da1f9e0d85/volumes"
Oct 31 14:01:14.837099 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:14.837065    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"]
Oct 31 14:01:14.848963 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:01:14.848908    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8f2762_7f5a_4bf8_b40f_981f560b4e28.slice/crio-e1f74ee07e1fa23f60e82fa16e6873edd89b3888c4ae93ec82e7acf314d127f4 WatchSource:0}: Error finding container e1f74ee07e1fa23f60e82fa16e6873edd89b3888c4ae93ec82e7acf314d127f4: Status 404 returned error can't find the container with id e1f74ee07e1fa23f60e82fa16e6873edd89b3888c4ae93ec82e7acf314d127f4
Oct 31 14:01:15.335420 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:15.335374    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:15.394938 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:15.394909    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl" event={"ID":"ed8f2762-7f5a-4bf8-b40f-981f560b4e28","Type":"ContainerStarted","Data":"e1f74ee07e1fa23f60e82fa16e6873edd89b3888c4ae93ec82e7acf314d127f4"}
Oct 31 14:01:15.644940 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:15.644828    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:15.845060 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:15.845018    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv" podUID="71b06a52-1d53-48e7-a5bd-b25f92a89e91" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:01:16.187138 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:16.187101    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:16.372308 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:16.372275    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:16.404289 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:16.404251    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl" event={"ID":"ed8f2762-7f5a-4bf8-b40f-981f560b4e28","Type":"ContainerStarted","Data":"d6d64fd454f4dabfbaf1894863dba1a684b33b0af296a905ef07b9ec02fcd058"}
Oct 31 14:01:16.404715 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:16.404391    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:01:16.426951 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:16.426848    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl" podStartSLOduration=1.918488359 podStartE2EDuration="2.42683308s" totalImagesPullingTime="508.344721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:01:16.425150428 +0000 UTC m=+3473.823499054" watchObservedRunningTime="2026-10-31 14:01:16.42683308 +0000 UTC m=+3473.825181705"
Oct 31 14:01:16.982496 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:16.982412    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:17.744597 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:17.744532    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:17.749377 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:17.749281    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"]
Oct 31 14:01:17.749574 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:17.749491    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc" podUID="2c55141e-2afe-4d0d-ba77-834eca70c27f" containerName="payload-pre-processing" containerID="cri-o://b1f27fb7566dc8d9f06da33b34381501ceaf33178ac7c3a9985c7841f1569683" gracePeriod=30
Oct 31 14:01:17.906849 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:17.906816    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:18.083807 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.080383    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:18.109364 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.109060    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"]
Oct 31 14:01:18.109589 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.109436    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9ca72-zf8n9"]
Oct 31 14:01:18.241752 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.241721    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:18.262625 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.262588    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"
Oct 31 14:01:18.406861 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.406635    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:18.423272 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.423232    4428 generic.go:356] "Generic (PLEG): container finished" podID="2c55141e-2afe-4d0d-ba77-834eca70c27f" containerID="b1f27fb7566dc8d9f06da33b34381501ceaf33178ac7c3a9985c7841f1569683" exitCode=0
Oct 31 14:01:18.423487 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.423287    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"
Oct 31 14:01:18.423487 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.423287    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc" event={"ID":"2c55141e-2afe-4d0d-ba77-834eca70c27f","Type":"ContainerDied","Data":"b1f27fb7566dc8d9f06da33b34381501ceaf33178ac7c3a9985c7841f1569683"}
Oct 31 14:01:18.423487 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.423362    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc" event={"ID":"2c55141e-2afe-4d0d-ba77-834eca70c27f","Type":"ContainerDied","Data":"1b47aefad773066867bb616c63eea5fd5d46462eff6285cb4f208b3ed722bad9"}
Oct 31 14:01:18.423487 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.423392    4428 scope.go:122] "RemoveContainer" containerID="b1f27fb7566dc8d9f06da33b34381501ceaf33178ac7c3a9985c7841f1569683"
Oct 31 14:01:18.436008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.435985    4428 scope.go:122] "RemoveContainer" containerID="b1f27fb7566dc8d9f06da33b34381501ceaf33178ac7c3a9985c7841f1569683"
Oct 31 14:01:18.436120 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.436050    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c55141e-2afe-4d0d-ba77-834eca70c27f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c55141e-2afe-4d0d-ba77-834eca70c27f-plugins-config-volume\") pod \"2c55141e-2afe-4d0d-ba77-834eca70c27f\" (UID: \"2c55141e-2afe-4d0d-ba77-834eca70c27f\") "
Oct 31 14:01:18.436272 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.436254    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c55141e-2afe-4d0d-ba77-834eca70c27f-kube-api-access-tt98t\" (UniqueName: \"kubernetes.io/projected/2c55141e-2afe-4d0d-ba77-834eca70c27f-kube-api-access-tt98t\") pod \"2c55141e-2afe-4d0d-ba77-834eca70c27f\" (UID: \"2c55141e-2afe-4d0d-ba77-834eca70c27f\") "
Oct 31 14:01:18.436394 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:01:18.436365    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f27fb7566dc8d9f06da33b34381501ceaf33178ac7c3a9985c7841f1569683\": container with ID starting with b1f27fb7566dc8d9f06da33b34381501ceaf33178ac7c3a9985c7841f1569683 not found: ID does not exist" containerID="b1f27fb7566dc8d9f06da33b34381501ceaf33178ac7c3a9985c7841f1569683"
Oct 31 14:01:18.436462 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.436406    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f27fb7566dc8d9f06da33b34381501ceaf33178ac7c3a9985c7841f1569683"} err="failed to get container status \"b1f27fb7566dc8d9f06da33b34381501ceaf33178ac7c3a9985c7841f1569683\": rpc error: code = NotFound desc = could not find container \"b1f27fb7566dc8d9f06da33b34381501ceaf33178ac7c3a9985c7841f1569683\": container with ID starting with b1f27fb7566dc8d9f06da33b34381501ceaf33178ac7c3a9985c7841f1569683 not found: ID does not exist"
Oct 31 14:01:18.436533 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.436509    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c55141e-2afe-4d0d-ba77-834eca70c27f-plugins-config-volume" pod "2c55141e-2afe-4d0d-ba77-834eca70c27f" (UID: "2c55141e-2afe-4d0d-ba77-834eca70c27f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:18.436973 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.436912    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c55141e-2afe-4d0d-ba77-834eca70c27f-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:18.438719 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.438697    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c55141e-2afe-4d0d-ba77-834eca70c27f-kube-api-access-tt98t" pod "2c55141e-2afe-4d0d-ba77-834eca70c27f" (UID: "2c55141e-2afe-4d0d-ba77-834eca70c27f"). InnerVolumeSpecName "kube-api-access-tt98t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:18.444555 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.444532    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"]
Oct 31 14:01:18.538258 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.538223    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt98t\" (UniqueName: \"kubernetes.io/projected/2c55141e-2afe-4d0d-ba77-834eca70c27f-kube-api-access-tt98t\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:18.564068 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.564013    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:18.752913 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.748298    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:18.774894 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.771279    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c4087d-1e63-4c89-b69b-52cfd37db57b" path="/var/lib/kubelet/pods/11c4087d-1e63-4c89-b69b-52cfd37db57b/volumes"
Oct 31 14:01:18.810907 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.810597    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"]
Oct 31 14:01:18.824112 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:18.821023    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f9ca72-7c4487d5f-86rtc"]
Oct 31 14:01:18.850144 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:01:18.849998    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:01:19.639786 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:19.639702    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:20.433535 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:20.433495    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:20.756850 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:20.756816    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c55141e-2afe-4d0d-ba77-834eca70c27f" path="/var/lib/kubelet/pods/2c55141e-2afe-4d0d-ba77-834eca70c27f/volumes"
Oct 31 14:01:21.260955 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:21.260455    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:21.928442 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:21.928413    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:22.130901 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:22.130256    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:22.287206 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:22.287122    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:22.420437 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:22.420405    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:01:22.728219 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:22.728186    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:23.074242 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.074151    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h" podUID="c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb" containerName="main" containerID="cri-o://65b31d3f2b141f856e6b740f24093e876378d9d2d88f522b396cb52651a1a506" gracePeriod=45
Oct 31 14:01:23.308972 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.308910    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:23.394749 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.394717    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 14:01:23.445665 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.445620    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs" podUID="6bfa6cd9-13c7-4201-9a1f-96877ae4288c" containerName="payload-processing" containerID="cri-o://5e133d39a91861a42f5cbeeff03d38c160409a1e61d226e4217d937ee98ae362" gracePeriod=25
Oct 31 14:01:23.468998 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.468959    4428 generic.go:356] "Generic (PLEG): container finished" podID="c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb" containerID="65b31d3f2b141f856e6b740f24093e876378d9d2d88f522b396cb52651a1a506" exitCode=0
Oct 31 14:01:23.469206 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.469016    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"
Oct 31 14:01:23.469206 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.469041    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h" event={"ID":"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb","Type":"ContainerDied","Data":"65b31d3f2b141f856e6b740f24093e876378d9d2d88f522b396cb52651a1a506"}
Oct 31 14:01:23.469206 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.469085    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h" event={"ID":"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb","Type":"ContainerDied","Data":"0bfa151a766353cb5cf5cdb92e92f88ad42ce16b5fe38be110b9643dd13cf28e"}
Oct 31 14:01:23.469206 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.469106    4428 scope.go:122] "RemoveContainer" containerID="65b31d3f2b141f856e6b740f24093e876378d9d2d88f522b396cb52651a1a506"
Oct 31 14:01:23.481417 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.481395    4428 scope.go:122] "RemoveContainer" containerID="65b31d3f2b141f856e6b740f24093e876378d9d2d88f522b396cb52651a1a506"
Oct 31 14:01:23.481760 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:01:23.481739    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b31d3f2b141f856e6b740f24093e876378d9d2d88f522b396cb52651a1a506\": container with ID starting with 65b31d3f2b141f856e6b740f24093e876378d9d2d88f522b396cb52651a1a506 not found: ID does not exist" containerID="65b31d3f2b141f856e6b740f24093e876378d9d2d88f522b396cb52651a1a506"
Oct 31 14:01:23.481808 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.481770    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b31d3f2b141f856e6b740f24093e876378d9d2d88f522b396cb52651a1a506"} err="failed to get container status \"65b31d3f2b141f856e6b740f24093e876378d9d2d88f522b396cb52651a1a506\": rpc error: code = NotFound desc = could not find container \"65b31d3f2b141f856e6b740f24093e876378d9d2d88f522b396cb52651a1a506\": container with ID starting with 65b31d3f2b141f856e6b740f24093e876378d9d2d88f522b396cb52651a1a506 not found: ID does not exist"
Oct 31 14:01:23.545900 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.545846    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-dshm\") pod \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") "
Oct 31 14:01:23.546076 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.545987    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-home\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-home\") pod \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") "
Oct 31 14:01:23.546146 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.546124    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-model-cache\") pod \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") "
Oct 31 14:01:23.546237 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.546212    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-home" pod "c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb" (UID: "c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:23.546304 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.546282    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-kube-api-access-vz92s\" (UniqueName: \"kubernetes.io/projected/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-kube-api-access-vz92s\") pod \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") "
Oct 31 14:01:23.546373 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.546334    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-tls-certs\") pod \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") "
Oct 31 14:01:23.546373 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.546356    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-model-cache" pod "c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb" (UID: "c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:23.546475 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.546367    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-tmp-dir\") pod \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\" (UID: \"c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb\") "
Oct 31 14:01:23.546605 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.546556    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-tmp-dir" pod "c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb" (UID: "c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:23.547079 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.547061    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:23.547167 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.547080    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:23.547167 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.547089    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:23.548613 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.548582    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-dshm" pod "c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb" (UID: "c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:23.549091 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.549067    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-tls-certs" pod "c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb" (UID: "c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:23.549177 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.549111    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-kube-api-access-vz92s" pod "c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb" (UID: "c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb"). InnerVolumeSpecName "kube-api-access-vz92s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:23.648287 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.648190    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz92s\" (UniqueName: \"kubernetes.io/projected/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-kube-api-access-vz92s\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:23.648287 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.648225    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:23.648287 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.648237    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:23.798575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.798540    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"]
Oct 31 14:01:23.808010 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:23.807981    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-14ba5a/rate-test-model-14ba5a-kserve-66d59c65d9-w279h"]
Oct 31 14:01:24.213894 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:24.209560    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:24.756693 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:24.756656    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb" path="/var/lib/kubelet/pods/c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb/volumes"
Oct 31 14:01:25.033080 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:25.032989    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:25.844707 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:25.844663    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv" podUID="71b06a52-1d53-48e7-a5bd-b25f92a89e91" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:01:25.893848 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:25.893797    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:26.049163 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:26.049129    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:26.138413 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:26.138271    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"]
Oct 31 14:01:26.138592 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:26.138562    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt" podUID="c2da70ae-abad-42d5-be21-0fd2b83848e0" containerName="istio-proxy" containerID="cri-o://9dbd592ed3119777464e88f2bad4f0d069d587b80fde38640207c8c951a7cc93" gracePeriod=30
Oct 31 14:01:26.224244 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:26.224208    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:27.242578 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:27.242509    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:27.973587 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:27.971916    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:28.404633 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:28.404545    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs" podUID="6bfa6cd9-13c7-4201-9a1f-96877ae4288c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:01:28.697050 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:28.696912    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:29.425035 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:29.424993    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:30.145147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:30.145060    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:30.341999 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:30.341959    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:30.444942 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:30.444819    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"]
Oct 31 14:01:30.445440 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:30.445171    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9" podUID="4608de93-17e1-4803-a234-bfb4e24e559f" containerName="istio-proxy" containerID="cri-o://0e0f127e614ce872d8061097c9eb740176d4be08ebfff71d0c4d52491bc13647" gracePeriod=30
Oct 31 14:01:30.519821 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:30.519781    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:30.697008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:30.696810    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:30.966357 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:30.966322    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:31.078116 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.078089    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"
Oct 31 14:01:31.169822 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.169790    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b06a52-1d53-48e7-a5bd-b25f92a89e91-kube-api-access-x7t52\" (UniqueName: \"kubernetes.io/projected/71b06a52-1d53-48e7-a5bd-b25f92a89e91-kube-api-access-x7t52\") pod \"71b06a52-1d53-48e7-a5bd-b25f92a89e91\" (UID: \"71b06a52-1d53-48e7-a5bd-b25f92a89e91\") "
Oct 31 14:01:31.170011 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.169990    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b06a52-1d53-48e7-a5bd-b25f92a89e91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b06a52-1d53-48e7-a5bd-b25f92a89e91-plugins-config-volume\") pod \"71b06a52-1d53-48e7-a5bd-b25f92a89e91\" (UID: \"71b06a52-1d53-48e7-a5bd-b25f92a89e91\") "
Oct 31 14:01:31.170362 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.170333    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b06a52-1d53-48e7-a5bd-b25f92a89e91-plugins-config-volume" pod "71b06a52-1d53-48e7-a5bd-b25f92a89e91" (UID: "71b06a52-1d53-48e7-a5bd-b25f92a89e91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:31.170637 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.170619    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b06a52-1d53-48e7-a5bd-b25f92a89e91-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:31.172946 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.172923    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b06a52-1d53-48e7-a5bd-b25f92a89e91-kube-api-access-x7t52" pod "71b06a52-1d53-48e7-a5bd-b25f92a89e91" (UID: "71b06a52-1d53-48e7-a5bd-b25f92a89e91"). InnerVolumeSpecName "kube-api-access-x7t52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:31.272507 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.272471    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7t52\" (UniqueName: \"kubernetes.io/projected/71b06a52-1d53-48e7-a5bd-b25f92a89e91-kube-api-access-x7t52\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:31.401970 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.401941    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 14:01:31.519392 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.519247    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:31.535818 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.535784    4428 generic.go:356] "Generic (PLEG): container finished" podID="71b06a52-1d53-48e7-a5bd-b25f92a89e91" containerID="8b699afc29f2d8d9241147d585c8407538f20577735eb3be8321b037c9fc1c3c" exitCode=0
Oct 31 14:01:31.536050 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.535842    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"
Oct 31 14:01:31.536050 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.535909    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv" event={"ID":"71b06a52-1d53-48e7-a5bd-b25f92a89e91","Type":"ContainerDied","Data":"8b699afc29f2d8d9241147d585c8407538f20577735eb3be8321b037c9fc1c3c"}
Oct 31 14:01:31.536050 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.535973    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv" event={"ID":"71b06a52-1d53-48e7-a5bd-b25f92a89e91","Type":"ContainerDied","Data":"99a6f8197c4f2b0e8483c0f509516d967181c01d55806d068c8440f002cbe2e8"}
Oct 31 14:01:31.536050 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.536007    4428 scope.go:122] "RemoveContainer" containerID="8b699afc29f2d8d9241147d585c8407538f20577735eb3be8321b037c9fc1c3c"
Oct 31 14:01:31.538299 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.538268    4428 generic.go:356] "Generic (PLEG): container finished" podID="c2da70ae-abad-42d5-be21-0fd2b83848e0" containerID="9dbd592ed3119777464e88f2bad4f0d069d587b80fde38640207c8c951a7cc93" exitCode=0
Oct 31 14:01:31.538478 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.538301    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt" event={"ID":"c2da70ae-abad-42d5-be21-0fd2b83848e0","Type":"ContainerDied","Data":"9dbd592ed3119777464e88f2bad4f0d069d587b80fde38640207c8c951a7cc93"}
Oct 31 14:01:31.538478 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.538334    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"
Oct 31 14:01:31.538478 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.538374    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt" event={"ID":"c2da70ae-abad-42d5-be21-0fd2b83848e0","Type":"ContainerDied","Data":"ce5f75d96d644a40dabddc77a5d53632b0cef229142249cf6e226403e98f1154"}
Oct 31 14:01:31.547731 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.547699    4428 scope.go:122] "RemoveContainer" containerID="8b699afc29f2d8d9241147d585c8407538f20577735eb3be8321b037c9fc1c3c"
Oct 31 14:01:31.548106 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:01:31.548085    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b699afc29f2d8d9241147d585c8407538f20577735eb3be8321b037c9fc1c3c\": container with ID starting with 8b699afc29f2d8d9241147d585c8407538f20577735eb3be8321b037c9fc1c3c not found: ID does not exist" containerID="8b699afc29f2d8d9241147d585c8407538f20577735eb3be8321b037c9fc1c3c"
Oct 31 14:01:31.548193 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.548136    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b699afc29f2d8d9241147d585c8407538f20577735eb3be8321b037c9fc1c3c"} err="failed to get container status \"8b699afc29f2d8d9241147d585c8407538f20577735eb3be8321b037c9fc1c3c\": rpc error: code = NotFound desc = could not find container \"8b699afc29f2d8d9241147d585c8407538f20577735eb3be8321b037c9fc1c3c\": container with ID starting with 8b699afc29f2d8d9241147d585c8407538f20577735eb3be8321b037c9fc1c3c not found: ID does not exist"
Oct 31 14:01:31.548193 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.548161    4428 scope.go:122] "RemoveContainer" containerID="9dbd592ed3119777464e88f2bad4f0d069d587b80fde38640207c8c951a7cc93"
Oct 31 14:01:31.559086 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.559065    4428 scope.go:122] "RemoveContainer" containerID="36e54d51c0030b7bffd4c8fe58c4d90e35019d908827fbbca0b30fa559097371"
Oct 31 14:01:31.566392 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.566363    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"]
Oct 31 14:01:31.570274 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.570249    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-599291-68bc448d7-rk4lv"]
Oct 31 14:01:31.571737 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.571716    4428 scope.go:122] "RemoveContainer" containerID="9dbd592ed3119777464e88f2bad4f0d069d587b80fde38640207c8c951a7cc93"
Oct 31 14:01:31.572153 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:01:31.572128    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dbd592ed3119777464e88f2bad4f0d069d587b80fde38640207c8c951a7cc93\": container with ID starting with 9dbd592ed3119777464e88f2bad4f0d069d587b80fde38640207c8c951a7cc93 not found: ID does not exist" containerID="9dbd592ed3119777464e88f2bad4f0d069d587b80fde38640207c8c951a7cc93"
Oct 31 14:01:31.572255 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.572160    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbd592ed3119777464e88f2bad4f0d069d587b80fde38640207c8c951a7cc93"} err="failed to get container status \"9dbd592ed3119777464e88f2bad4f0d069d587b80fde38640207c8c951a7cc93\": rpc error: code = NotFound desc = could not find container \"9dbd592ed3119777464e88f2bad4f0d069d587b80fde38640207c8c951a7cc93\": container with ID starting with 9dbd592ed3119777464e88f2bad4f0d069d587b80fde38640207c8c951a7cc93 not found: ID does not exist"
Oct 31 14:01:31.572255 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.572179    4428 scope.go:122] "RemoveContainer" containerID="36e54d51c0030b7bffd4c8fe58c4d90e35019d908827fbbca0b30fa559097371"
Oct 31 14:01:31.572452 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:01:31.572435    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36e54d51c0030b7bffd4c8fe58c4d90e35019d908827fbbca0b30fa559097371\": container with ID starting with 36e54d51c0030b7bffd4c8fe58c4d90e35019d908827fbbca0b30fa559097371 not found: ID does not exist" containerID="36e54d51c0030b7bffd4c8fe58c4d90e35019d908827fbbca0b30fa559097371"
Oct 31 14:01:31.572514 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.572456    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e54d51c0030b7bffd4c8fe58c4d90e35019d908827fbbca0b30fa559097371"} err="failed to get container status \"36e54d51c0030b7bffd4c8fe58c4d90e35019d908827fbbca0b30fa559097371\": rpc error: code = NotFound desc = could not find container \"36e54d51c0030b7bffd4c8fe58c4d90e35019d908827fbbca0b30fa559097371\": container with ID starting with 36e54d51c0030b7bffd4c8fe58c4d90e35019d908827fbbca0b30fa559097371 not found: ID does not exist"
Oct 31 14:01:31.576953 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.576920    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-data\") pod \"c2da70ae-abad-42d5-be21-0fd2b83848e0\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") "
Oct 31 14:01:31.577066 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.576967    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-envoy\") pod \"c2da70ae-abad-42d5-be21-0fd2b83848e0\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") "
Oct 31 14:01:31.577066 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.577059    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-podinfo\") pod \"c2da70ae-abad-42d5-be21-0fd2b83848e0\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") "
Oct 31 14:01:31.577175 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.577109    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-workload-socket\") pod \"c2da70ae-abad-42d5-be21-0fd2b83848e0\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") "
Oct 31 14:01:31.577175 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.577169    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2da70ae-abad-42d5-be21-0fd2b83848e0-kube-api-access-qrlpd\" (UniqueName: \"kubernetes.io/projected/c2da70ae-abad-42d5-be21-0fd2b83848e0-kube-api-access-qrlpd\") pod \"c2da70ae-abad-42d5-be21-0fd2b83848e0\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") "
Oct 31 14:01:31.577290 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.577176    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-data" pod "c2da70ae-abad-42d5-be21-0fd2b83848e0" (UID: "c2da70ae-abad-42d5-be21-0fd2b83848e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:31.577290 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.577246    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-token\" (UniqueName: \"kubernetes.io/projected/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-token\") pod \"c2da70ae-abad-42d5-be21-0fd2b83848e0\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") "
Oct 31 14:01:31.577405 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.577286    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-workload-certs\") pod \"c2da70ae-abad-42d5-be21-0fd2b83848e0\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") "
Oct 31 14:01:31.577405 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.577395    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-credential-socket\") pod \"c2da70ae-abad-42d5-be21-0fd2b83848e0\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") "
Oct 31 14:01:31.577516 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.577445    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-workload-socket" pod "c2da70ae-abad-42d5-be21-0fd2b83848e0" (UID: "c2da70ae-abad-42d5-be21-0fd2b83848e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:31.577644 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.577535    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2da70ae-abad-42d5-be21-0fd2b83848e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2da70ae-abad-42d5-be21-0fd2b83848e0-istiod-ca-cert\") pod \"c2da70ae-abad-42d5-be21-0fd2b83848e0\" (UID: \"c2da70ae-abad-42d5-be21-0fd2b83848e0\") "
Oct 31 14:01:31.577644 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.577579    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-workload-certs" pod "c2da70ae-abad-42d5-be21-0fd2b83848e0" (UID: "c2da70ae-abad-42d5-be21-0fd2b83848e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:31.577753 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.577701    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-credential-socket" pod "c2da70ae-abad-42d5-be21-0fd2b83848e0" (UID: "c2da70ae-abad-42d5-be21-0fd2b83848e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:31.577998 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.577975    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2da70ae-abad-42d5-be21-0fd2b83848e0-istiod-ca-cert" pod "c2da70ae-abad-42d5-be21-0fd2b83848e0" (UID: "c2da70ae-abad-42d5-be21-0fd2b83848e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:31.578310 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.578291    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2da70ae-abad-42d5-be21-0fd2b83848e0-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:31.578369 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.578317    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:31.578369 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.578332    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:31.578369 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.578345    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:31.578533 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.578358    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:31.580138 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.580105    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-envoy" pod "c2da70ae-abad-42d5-be21-0fd2b83848e0" (UID: "c2da70ae-abad-42d5-be21-0fd2b83848e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:31.580233 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.580175    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-token" pod "c2da70ae-abad-42d5-be21-0fd2b83848e0" (UID: "c2da70ae-abad-42d5-be21-0fd2b83848e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:31.580359 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.580329    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2da70ae-abad-42d5-be21-0fd2b83848e0-kube-api-access-qrlpd" pod "c2da70ae-abad-42d5-be21-0fd2b83848e0" (UID: "c2da70ae-abad-42d5-be21-0fd2b83848e0"). InnerVolumeSpecName "kube-api-access-qrlpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:31.580903 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.580858    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-podinfo" pod "c2da70ae-abad-42d5-be21-0fd2b83848e0" (UID: "c2da70ae-abad-42d5-be21-0fd2b83848e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:01:31.680045 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.679994    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:31.680045 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.680038    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrlpd\" (UniqueName: \"kubernetes.io/projected/c2da70ae-abad-42d5-be21-0fd2b83848e0-kube-api-access-qrlpd\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:31.680045 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.680050    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:31.680045 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.680067    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2da70ae-abad-42d5-be21-0fd2b83848e0-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:31.867847 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.867813    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"]
Oct 31 14:01:31.874122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:31.874088    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9ca72-openshift-default-5c499594fc-5hwkt"]
Oct 31 14:01:32.228344 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.228312    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"]
Oct 31 14:01:32.228592 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.228563    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps" podUID="75240edd-b28b-4986-992b-6557ad0f41be" containerName="maas-api" containerID="cri-o://51b1ab3e2c4a5ffdfa9097a7a00af21bee7d3d146684595f0d07b934b6d9d985" gracePeriod=30
Oct 31 14:01:32.328180 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.328147    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:32.486997 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.486909    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"]
Oct 31 14:01:32.487248 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.487219    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b" podUID="66b6b2a9-e6b8-4848-9fb7-4886a7db34e4" containerName="payload-processing" containerID="cri-o://efbf06733dc72043c1bb4e624496746c203f071856e093c08cec16fb2ad5240d" gracePeriod=30
Oct 31 14:01:32.493089 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.492235    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"]
Oct 31 14:01:32.493089 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.492737    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg" podUID="9825f1d9-236b-4516-94b1-058e389a702c" containerName="payload-pre-processing" containerID="cri-o://9b6082754709088e13b8f7e33021f896bdfc3e0cee584aa229c298e9e0d9959b" gracePeriod=30
Oct 31 14:01:32.497143 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.496174    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:32.522409 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.520478    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"]
Oct 31 14:01:32.559456 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.559421    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-974711-nd56l"]
Oct 31 14:01:32.569924 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.569885    4428 generic.go:356] "Generic (PLEG): container finished" podID="75240edd-b28b-4986-992b-6557ad0f41be" containerID="51b1ab3e2c4a5ffdfa9097a7a00af21bee7d3d146684595f0d07b934b6d9d985" exitCode=0
Oct 31 14:01:32.570090 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.569982    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps" event={"ID":"75240edd-b28b-4986-992b-6557ad0f41be","Type":"ContainerDied","Data":"51b1ab3e2c4a5ffdfa9097a7a00af21bee7d3d146684595f0d07b934b6d9d985"}
Oct 31 14:01:32.646614 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.646587    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 14:01:32.656107 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.656073    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:32.832827 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.803698    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75240edd-b28b-4986-992b-6557ad0f41be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75240edd-b28b-4986-992b-6557ad0f41be-metrics-certs\") pod \"75240edd-b28b-4986-992b-6557ad0f41be\" (UID: \"75240edd-b28b-4986-992b-6557ad0f41be\") "
Oct 31 14:01:32.832827 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.803745    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75240edd-b28b-4986-992b-6557ad0f41be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75240edd-b28b-4986-992b-6557ad0f41be-maas-api-tls\") pod \"75240edd-b28b-4986-992b-6557ad0f41be\" (UID: \"75240edd-b28b-4986-992b-6557ad0f41be\") "
Oct 31 14:01:32.832827 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.803832    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75240edd-b28b-4986-992b-6557ad0f41be-kube-api-access-dnlz8\" (UniqueName: \"kubernetes.io/projected/75240edd-b28b-4986-992b-6557ad0f41be-kube-api-access-dnlz8\") pod \"75240edd-b28b-4986-992b-6557ad0f41be\" (UID: \"75240edd-b28b-4986-992b-6557ad0f41be\") "
Oct 31 14:01:32.832827 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.804008    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75240edd-b28b-4986-992b-6557ad0f41be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75240edd-b28b-4986-992b-6557ad0f41be-openshift-service-ca\") pod \"75240edd-b28b-4986-992b-6557ad0f41be\" (UID: \"75240edd-b28b-4986-992b-6557ad0f41be\") "
Oct 31 14:01:32.832827 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.806939    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75240edd-b28b-4986-992b-6557ad0f41be-openshift-service-ca" pod "75240edd-b28b-4986-992b-6557ad0f41be" (UID: "75240edd-b28b-4986-992b-6557ad0f41be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:32.832827 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.813310    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="570a38b1-3f0e-4a64-a66b-24786d0813a0" path="/var/lib/kubelet/pods/570a38b1-3f0e-4a64-a66b-24786d0813a0/volumes"
Oct 31 14:01:32.832827 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.813848    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b06a52-1d53-48e7-a5bd-b25f92a89e91" path="/var/lib/kubelet/pods/71b06a52-1d53-48e7-a5bd-b25f92a89e91/volumes"
Oct 31 14:01:32.832827 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.814360    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2da70ae-abad-42d5-be21-0fd2b83848e0" path="/var/lib/kubelet/pods/c2da70ae-abad-42d5-be21-0fd2b83848e0/volumes"
Oct 31 14:01:32.832827 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.819338    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75240edd-b28b-4986-992b-6557ad0f41be-metrics-certs" pod "75240edd-b28b-4986-992b-6557ad0f41be" (UID: "75240edd-b28b-4986-992b-6557ad0f41be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:32.832827 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.819385    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75240edd-b28b-4986-992b-6557ad0f41be-kube-api-access-dnlz8" pod "75240edd-b28b-4986-992b-6557ad0f41be" (UID: "75240edd-b28b-4986-992b-6557ad0f41be"). InnerVolumeSpecName "kube-api-access-dnlz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:32.832827 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.824569    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75240edd-b28b-4986-992b-6557ad0f41be-maas-api-tls" pod "75240edd-b28b-4986-992b-6557ad0f41be" (UID: "75240edd-b28b-4986-992b-6557ad0f41be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:32.906531 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.906481    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75240edd-b28b-4986-992b-6557ad0f41be-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:32.906531 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.906534    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75240edd-b28b-4986-992b-6557ad0f41be-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:32.908094 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.906549    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75240edd-b28b-4986-992b-6557ad0f41be-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:32.908094 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.906563    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnlz8\" (UniqueName: \"kubernetes.io/projected/75240edd-b28b-4986-992b-6557ad0f41be-kube-api-access-dnlz8\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:32.969186 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.969130    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:32.983412 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:32.983134    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"
Oct 31 14:01:33.026826 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.026791    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"
Oct 31 14:01:33.110400 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.110182    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9825f1d9-236b-4516-94b1-058e389a702c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9825f1d9-236b-4516-94b1-058e389a702c-plugins-config-volume\") pod \"9825f1d9-236b-4516-94b1-058e389a702c\" (UID: \"9825f1d9-236b-4516-94b1-058e389a702c\") "
Oct 31 14:01:33.110400 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.110261    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b6b2a9-e6b8-4848-9fb7-4886a7db34e4-kube-api-access-bss4h\" (UniqueName: \"kubernetes.io/projected/66b6b2a9-e6b8-4848-9fb7-4886a7db34e4-kube-api-access-bss4h\") pod \"66b6b2a9-e6b8-4848-9fb7-4886a7db34e4\" (UID: \"66b6b2a9-e6b8-4848-9fb7-4886a7db34e4\") "
Oct 31 14:01:33.110647 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.110521    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66b6b2a9-e6b8-4848-9fb7-4886a7db34e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b6b2a9-e6b8-4848-9fb7-4886a7db34e4-plugins-config-volume\") pod \"66b6b2a9-e6b8-4848-9fb7-4886a7db34e4\" (UID: \"66b6b2a9-e6b8-4848-9fb7-4886a7db34e4\") "
Oct 31 14:01:33.110647 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.110625    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9825f1d9-236b-4516-94b1-058e389a702c-kube-api-access-4mkjr\" (UniqueName: \"kubernetes.io/projected/9825f1d9-236b-4516-94b1-058e389a702c-kube-api-access-4mkjr\") pod \"9825f1d9-236b-4516-94b1-058e389a702c\" (UID: \"9825f1d9-236b-4516-94b1-058e389a702c\") "
Oct 31 14:01:33.111043 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.111011    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b6b2a9-e6b8-4848-9fb7-4886a7db34e4-plugins-config-volume" pod "66b6b2a9-e6b8-4848-9fb7-4886a7db34e4" (UID: "66b6b2a9-e6b8-4848-9fb7-4886a7db34e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:33.111257 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.111242    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b6b2a9-e6b8-4848-9fb7-4886a7db34e4-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:33.111617 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.111591    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9825f1d9-236b-4516-94b1-058e389a702c-plugins-config-volume" pod "9825f1d9-236b-4516-94b1-058e389a702c" (UID: "9825f1d9-236b-4516-94b1-058e389a702c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:33.113265 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.113197    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9825f1d9-236b-4516-94b1-058e389a702c-kube-api-access-4mkjr" pod "9825f1d9-236b-4516-94b1-058e389a702c" (UID: "9825f1d9-236b-4516-94b1-058e389a702c"). InnerVolumeSpecName "kube-api-access-4mkjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:33.113698 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.113678    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b6b2a9-e6b8-4848-9fb7-4886a7db34e4-kube-api-access-bss4h" pod "66b6b2a9-e6b8-4848-9fb7-4886a7db34e4" (UID: "66b6b2a9-e6b8-4848-9fb7-4886a7db34e4"). InnerVolumeSpecName "kube-api-access-bss4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:33.140601 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.140557    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:33.215782 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.215693    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9825f1d9-236b-4516-94b1-058e389a702c-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:33.215782 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.215737    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bss4h\" (UniqueName: \"kubernetes.io/projected/66b6b2a9-e6b8-4848-9fb7-4886a7db34e4-kube-api-access-bss4h\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:33.215782 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.215753    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mkjr\" (UniqueName: \"kubernetes.io/projected/9825f1d9-236b-4516-94b1-058e389a702c-kube-api-access-4mkjr\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:33.343118 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.343082    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:33.579010 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.578945    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps" event={"ID":"75240edd-b28b-4986-992b-6557ad0f41be","Type":"ContainerDied","Data":"f4025b6a5f0669236d4d6663f7e7fc6022a2769ab02cd0c4a985a3835cc71514"}
Oct 31 14:01:33.579513 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.579047    4428 scope.go:122] "RemoveContainer" containerID="51b1ab3e2c4a5ffdfa9097a7a00af21bee7d3d146684595f0d07b934b6d9d985"
Oct 31 14:01:33.579666 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.579637    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"
Oct 31 14:01:33.580731 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.580708    4428 generic.go:356] "Generic (PLEG): container finished" podID="9825f1d9-236b-4516-94b1-058e389a702c" containerID="9b6082754709088e13b8f7e33021f896bdfc3e0cee584aa229c298e9e0d9959b" exitCode=0
Oct 31 14:01:33.580855 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.580838    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg" event={"ID":"9825f1d9-236b-4516-94b1-058e389a702c","Type":"ContainerDied","Data":"9b6082754709088e13b8f7e33021f896bdfc3e0cee584aa229c298e9e0d9959b"}
Oct 31 14:01:33.580964 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.580925    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg" event={"ID":"9825f1d9-236b-4516-94b1-058e389a702c","Type":"ContainerDied","Data":"c1c31dcebf4653ad04b4e1f27a6043a7063986a342db694da98fd6da1112c638"}
Oct 31 14:01:33.581028 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.580998    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"
Oct 31 14:01:33.585332 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.585311    4428 generic.go:356] "Generic (PLEG): container finished" podID="66b6b2a9-e6b8-4848-9fb7-4886a7db34e4" containerID="efbf06733dc72043c1bb4e624496746c203f071856e093c08cec16fb2ad5240d" exitCode=0
Oct 31 14:01:33.585447 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.585361    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"
Oct 31 14:01:33.585447 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.585400    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b" event={"ID":"66b6b2a9-e6b8-4848-9fb7-4886a7db34e4","Type":"ContainerDied","Data":"efbf06733dc72043c1bb4e624496746c203f071856e093c08cec16fb2ad5240d"}
Oct 31 14:01:33.585589 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.585446    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b" event={"ID":"66b6b2a9-e6b8-4848-9fb7-4886a7db34e4","Type":"ContainerDied","Data":"c05fa7d2d8f7f587b7c3014b7905e918311c21cafd05aa2d164f2f88f8bc015f"}
Oct 31 14:01:33.592181 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.592155    4428 scope.go:122] "RemoveContainer" containerID="9b6082754709088e13b8f7e33021f896bdfc3e0cee584aa229c298e9e0d9959b"
Oct 31 14:01:33.603289 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.603256    4428 scope.go:122] "RemoveContainer" containerID="9b6082754709088e13b8f7e33021f896bdfc3e0cee584aa229c298e9e0d9959b"
Oct 31 14:01:33.603622 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:01:33.603602    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b6082754709088e13b8f7e33021f896bdfc3e0cee584aa229c298e9e0d9959b\": container with ID starting with 9b6082754709088e13b8f7e33021f896bdfc3e0cee584aa229c298e9e0d9959b not found: ID does not exist" containerID="9b6082754709088e13b8f7e33021f896bdfc3e0cee584aa229c298e9e0d9959b"
Oct 31 14:01:33.603704 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.603633    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b6082754709088e13b8f7e33021f896bdfc3e0cee584aa229c298e9e0d9959b"} err="failed to get container status \"9b6082754709088e13b8f7e33021f896bdfc3e0cee584aa229c298e9e0d9959b\": rpc error: code = NotFound desc = could not find container \"9b6082754709088e13b8f7e33021f896bdfc3e0cee584aa229c298e9e0d9959b\": container with ID starting with 9b6082754709088e13b8f7e33021f896bdfc3e0cee584aa229c298e9e0d9959b not found: ID does not exist"
Oct 31 14:01:33.603704 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.603653    4428 scope.go:122] "RemoveContainer" containerID="efbf06733dc72043c1bb4e624496746c203f071856e093c08cec16fb2ad5240d"
Oct 31 14:01:33.613599 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.613481    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"]
Oct 31 14:01:33.618174 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.618137    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-55687f99-qwrlg"]
Oct 31 14:01:33.619767 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.619625    4428 scope.go:122] "RemoveContainer" containerID="efbf06733dc72043c1bb4e624496746c203f071856e093c08cec16fb2ad5240d"
Oct 31 14:01:33.620083 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:01:33.620052    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efbf06733dc72043c1bb4e624496746c203f071856e093c08cec16fb2ad5240d\": container with ID starting with efbf06733dc72043c1bb4e624496746c203f071856e093c08cec16fb2ad5240d not found: ID does not exist" containerID="efbf06733dc72043c1bb4e624496746c203f071856e093c08cec16fb2ad5240d"
Oct 31 14:01:33.620168 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.620092    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbf06733dc72043c1bb4e624496746c203f071856e093c08cec16fb2ad5240d"} err="failed to get container status \"efbf06733dc72043c1bb4e624496746c203f071856e093c08cec16fb2ad5240d\": rpc error: code = NotFound desc = could not find container \"efbf06733dc72043c1bb4e624496746c203f071856e093c08cec16fb2ad5240d\": container with ID starting with efbf06733dc72043c1bb4e624496746c203f071856e093c08cec16fb2ad5240d not found: ID does not exist"
Oct 31 14:01:33.630922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.630891    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"]
Oct 31 14:01:33.636241 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.636212    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-974711-7d99fc5fc8-kbcps"]
Oct 31 14:01:33.649145 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.649116    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"]
Oct 31 14:01:33.666427 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.666397    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-403dff-7f4fb76d66-gd69b"]
Oct 31 14:01:33.669620 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.669585    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:33.922473 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.922373    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"]
Oct 31 14:01:33.938540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:33.938499    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"]
Oct 31 14:01:34.058371 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:34.058337    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:34.230828 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:34.230796    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:34.496858 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:34.496766    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:34.757340 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:34.757261    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b6b2a9-e6b8-4848-9fb7-4886a7db34e4" path="/var/lib/kubelet/pods/66b6b2a9-e6b8-4848-9fb7-4886a7db34e4/volumes"
Oct 31 14:01:34.757835 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:34.757778    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75240edd-b28b-4986-992b-6557ad0f41be" path="/var/lib/kubelet/pods/75240edd-b28b-4986-992b-6557ad0f41be/volumes"
Oct 31 14:01:34.758221 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:34.758207    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9825f1d9-236b-4516-94b1-058e389a702c" path="/var/lib/kubelet/pods/9825f1d9-236b-4516-94b1-058e389a702c/volumes"
Oct 31 14:01:35.084238 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.084145    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:35.606969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.606936    4428 generic.go:356] "Generic (PLEG): container finished" podID="4608de93-17e1-4803-a234-bfb4e24e559f" containerID="0e0f127e614ce872d8061097c9eb740176d4be08ebfff71d0c4d52491bc13647" exitCode=0
Oct 31 14:01:35.607194 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.607100    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9" event={"ID":"4608de93-17e1-4803-a234-bfb4e24e559f","Type":"ContainerDied","Data":"0e0f127e614ce872d8061097c9eb740176d4be08ebfff71d0c4d52491bc13647"}
Oct 31 14:01:35.607194 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.607150    4428 scope.go:122] "RemoveContainer" containerID="20dc94b7e65fdb678e77e620cd0146015123b97adce99691977fa9c6063f28a4"
Oct 31 14:01:35.739070 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.739038    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 14:01:35.825426 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.825379    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:35.855836 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.855795    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-workload-socket\") pod \"4608de93-17e1-4803-a234-bfb4e24e559f\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") "
Oct 31 14:01:35.855836 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.855834    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-credential-socket\") pod \"4608de93-17e1-4803-a234-bfb4e24e559f\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") "
Oct 31 14:01:35.856152 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.855910    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4608de93-17e1-4803-a234-bfb4e24e559f-istio-token\" (UniqueName: \"kubernetes.io/projected/4608de93-17e1-4803-a234-bfb4e24e559f-istio-token\") pod \"4608de93-17e1-4803-a234-bfb4e24e559f\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") "
Oct 31 14:01:35.856152 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.855960    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4608de93-17e1-4803-a234-bfb4e24e559f-kube-api-access-7fghl\" (UniqueName: \"kubernetes.io/projected/4608de93-17e1-4803-a234-bfb4e24e559f-kube-api-access-7fghl\") pod \"4608de93-17e1-4803-a234-bfb4e24e559f\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") "
Oct 31 14:01:35.856152 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.856094    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-workload-certs\") pod \"4608de93-17e1-4803-a234-bfb4e24e559f\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") "
Oct 31 14:01:35.856349 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.856144    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-workload-socket" pod "4608de93-17e1-4803-a234-bfb4e24e559f" (UID: "4608de93-17e1-4803-a234-bfb4e24e559f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:35.856349 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.856179    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-istio-envoy\") pod \"4608de93-17e1-4803-a234-bfb4e24e559f\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") "
Oct 31 14:01:35.856349 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.856260    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-credential-socket" pod "4608de93-17e1-4803-a234-bfb4e24e559f" (UID: "4608de93-17e1-4803-a234-bfb4e24e559f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:35.856349 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.856278    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4608de93-17e1-4803-a234-bfb4e24e559f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4608de93-17e1-4803-a234-bfb4e24e559f-istiod-ca-cert\") pod \"4608de93-17e1-4803-a234-bfb4e24e559f\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") "
Oct 31 14:01:35.856349 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.856331    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4608de93-17e1-4803-a234-bfb4e24e559f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4608de93-17e1-4803-a234-bfb4e24e559f-istio-podinfo\") pod \"4608de93-17e1-4803-a234-bfb4e24e559f\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") "
Oct 31 14:01:35.856632 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.856387    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-istio-data\") pod \"4608de93-17e1-4803-a234-bfb4e24e559f\" (UID: \"4608de93-17e1-4803-a234-bfb4e24e559f\") "
Oct 31 14:01:35.856632 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.856449    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-workload-certs" pod "4608de93-17e1-4803-a234-bfb4e24e559f" (UID: "4608de93-17e1-4803-a234-bfb4e24e559f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:35.857527 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.856796    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4608de93-17e1-4803-a234-bfb4e24e559f-istiod-ca-cert" pod "4608de93-17e1-4803-a234-bfb4e24e559f" (UID: "4608de93-17e1-4803-a234-bfb4e24e559f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:35.857527 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.856800    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-istio-data" pod "4608de93-17e1-4803-a234-bfb4e24e559f" (UID: "4608de93-17e1-4803-a234-bfb4e24e559f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:35.857527 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.857184    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4608de93-17e1-4803-a234-bfb4e24e559f-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:35.857527 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.857209    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:35.857527 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.857225    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:35.857527 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.857324    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:35.857527 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.857497    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:35.859277 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.859210    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4608de93-17e1-4803-a234-bfb4e24e559f-istio-token" pod "4608de93-17e1-4803-a234-bfb4e24e559f" (UID: "4608de93-17e1-4803-a234-bfb4e24e559f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:35.859402 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.859285    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-istio-envoy" pod "4608de93-17e1-4803-a234-bfb4e24e559f" (UID: "4608de93-17e1-4803-a234-bfb4e24e559f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:35.859467 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.859415    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4608de93-17e1-4803-a234-bfb4e24e559f-kube-api-access-7fghl" pod "4608de93-17e1-4803-a234-bfb4e24e559f" (UID: "4608de93-17e1-4803-a234-bfb4e24e559f"). InnerVolumeSpecName "kube-api-access-7fghl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:35.860100 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.860082    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4608de93-17e1-4803-a234-bfb4e24e559f-istio-podinfo" pod "4608de93-17e1-4803-a234-bfb4e24e559f" (UID: "4608de93-17e1-4803-a234-bfb4e24e559f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:01:35.937058 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.937017    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"]
Oct 31 14:01:35.958610 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.958518    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4608de93-17e1-4803-a234-bfb4e24e559f-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:35.958610 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.958552    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fghl\" (UniqueName: \"kubernetes.io/projected/4608de93-17e1-4803-a234-bfb4e24e559f-kube-api-access-7fghl\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:35.958610 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.958570    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4608de93-17e1-4803-a234-bfb4e24e559f-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:35.958610 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:35.958579    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4608de93-17e1-4803-a234-bfb4e24e559f-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:36.099573 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:36.099540    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:36.615539 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:36.615506    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"
Oct 31 14:01:36.615539 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:36.615532    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9" event={"ID":"4608de93-17e1-4803-a234-bfb4e24e559f","Type":"ContainerDied","Data":"ff698a69b53a75d8b7707e36a502318689cfcf387f6f185da15890fe709fe36e"}
Oct 31 14:01:36.615826 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:36.615613    4428 scope.go:122] "RemoveContainer" containerID="0e0f127e614ce872d8061097c9eb740176d4be08ebfff71d0c4d52491bc13647"
Oct 31 14:01:36.642064 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:36.642033    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"]
Oct 31 14:01:36.651663 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:36.651636    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-n4zh9"]
Oct 31 14:01:36.709828 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:36.698922    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"]
Oct 31 14:01:36.759148 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:36.759112    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4608de93-17e1-4803-a234-bfb4e24e559f" path="/var/lib/kubelet/pods/4608de93-17e1-4803-a234-bfb4e24e559f/volumes"
Oct 31 14:01:36.813155 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:36.813115    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:36.978909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:36.970515    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:37.707916 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:37.707734    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:38.404597 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:38.404556    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs" podUID="6bfa6cd9-13c7-4201-9a1f-96877ae4288c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:01:38.487280 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:38.487248    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:38.922749 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:38.922704    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc" podUID="63f881f7-c955-411e-8c47-f87e0c94b581" containerName="payload-processing" containerID="cri-o://90e8601b0f8153fdc7322c0f79b91ee8c65bb789d22833a17e893d2e0bbd58bd" gracePeriod=25
Oct 31 14:01:38.938830 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:38.938794    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8" podUID="335e2046-6ac2-4332-a3d3-b721751681df" containerName="payload-pre-processing" containerID="cri-o://72c31c199f1ced1b81b30de9b02871d16bd06f3cf4225d41efd2d26897a3a776" gracePeriod=25
Oct 31 14:01:39.270028 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.269999    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"
Oct 31 14:01:39.274063 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.274042    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"
Oct 31 14:01:39.380914 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.380841    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:39.426824 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.426765    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/335e2046-6ac2-4332-a3d3-b721751681df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335e2046-6ac2-4332-a3d3-b721751681df-plugins-config-volume\") pod \"335e2046-6ac2-4332-a3d3-b721751681df\" (UID: \"335e2046-6ac2-4332-a3d3-b721751681df\") "
Oct 31 14:01:39.427625 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.427561    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335e2046-6ac2-4332-a3d3-b721751681df-kube-api-access-fr6r7\" (UniqueName: \"kubernetes.io/projected/335e2046-6ac2-4332-a3d3-b721751681df-kube-api-access-fr6r7\") pod \"335e2046-6ac2-4332-a3d3-b721751681df\" (UID: \"335e2046-6ac2-4332-a3d3-b721751681df\") "
Oct 31 14:01:39.428038 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.427887    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f881f7-c955-411e-8c47-f87e0c94b581-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f881f7-c955-411e-8c47-f87e0c94b581-plugins-config-volume\") pod \"63f881f7-c955-411e-8c47-f87e0c94b581\" (UID: \"63f881f7-c955-411e-8c47-f87e0c94b581\") "
Oct 31 14:01:39.428290 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.428270    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f881f7-c955-411e-8c47-f87e0c94b581-kube-api-access-vwcsj\" (UniqueName: \"kubernetes.io/projected/63f881f7-c955-411e-8c47-f87e0c94b581-kube-api-access-vwcsj\") pod \"63f881f7-c955-411e-8c47-f87e0c94b581\" (UID: \"63f881f7-c955-411e-8c47-f87e0c94b581\") "
Oct 31 14:01:39.430030 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.429993    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335e2046-6ac2-4332-a3d3-b721751681df-plugins-config-volume" pod "335e2046-6ac2-4332-a3d3-b721751681df" (UID: "335e2046-6ac2-4332-a3d3-b721751681df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:39.431264 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.431231    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f881f7-c955-411e-8c47-f87e0c94b581-plugins-config-volume" pod "63f881f7-c955-411e-8c47-f87e0c94b581" (UID: "63f881f7-c955-411e-8c47-f87e0c94b581"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:39.434454 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.433254    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335e2046-6ac2-4332-a3d3-b721751681df-kube-api-access-fr6r7" pod "335e2046-6ac2-4332-a3d3-b721751681df" (UID: "335e2046-6ac2-4332-a3d3-b721751681df"). InnerVolumeSpecName "kube-api-access-fr6r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:39.442457 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.438562    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f881f7-c955-411e-8c47-f87e0c94b581-kube-api-access-vwcsj" pod "63f881f7-c955-411e-8c47-f87e0c94b581" (UID: "63f881f7-c955-411e-8c47-f87e0c94b581"). InnerVolumeSpecName "kube-api-access-vwcsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:39.530942 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.530842    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f881f7-c955-411e-8c47-f87e0c94b581-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:39.530942 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.530895    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwcsj\" (UniqueName: \"kubernetes.io/projected/63f881f7-c955-411e-8c47-f87e0c94b581-kube-api-access-vwcsj\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:39.530942 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.530906    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335e2046-6ac2-4332-a3d3-b721751681df-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:39.530942 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.530916    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr6r7\" (UniqueName: \"kubernetes.io/projected/335e2046-6ac2-4332-a3d3-b721751681df-kube-api-access-fr6r7\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:39.655344 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.655307    4428 generic.go:356] "Generic (PLEG): container finished" podID="335e2046-6ac2-4332-a3d3-b721751681df" containerID="72c31c199f1ced1b81b30de9b02871d16bd06f3cf4225d41efd2d26897a3a776" exitCode=0
Oct 31 14:01:39.655557 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.655366    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"
Oct 31 14:01:39.655557 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.655396    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8" event={"ID":"335e2046-6ac2-4332-a3d3-b721751681df","Type":"ContainerDied","Data":"72c31c199f1ced1b81b30de9b02871d16bd06f3cf4225d41efd2d26897a3a776"}
Oct 31 14:01:39.655557 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.655461    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8" event={"ID":"335e2046-6ac2-4332-a3d3-b721751681df","Type":"ContainerDied","Data":"7e656d7486656519199f73bc1d787508bed0527f0665bf5e5ea4d09fc7816872"}
Oct 31 14:01:39.655557 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.655492    4428 scope.go:122] "RemoveContainer" containerID="72c31c199f1ced1b81b30de9b02871d16bd06f3cf4225d41efd2d26897a3a776"
Oct 31 14:01:39.657175 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.657138    4428 generic.go:356] "Generic (PLEG): container finished" podID="63f881f7-c955-411e-8c47-f87e0c94b581" containerID="90e8601b0f8153fdc7322c0f79b91ee8c65bb789d22833a17e893d2e0bbd58bd" exitCode=0
Oct 31 14:01:39.657326 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.657172    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc" event={"ID":"63f881f7-c955-411e-8c47-f87e0c94b581","Type":"ContainerDied","Data":"90e8601b0f8153fdc7322c0f79b91ee8c65bb789d22833a17e893d2e0bbd58bd"}
Oct 31 14:01:39.657326 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.657189    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"
Oct 31 14:01:39.657326 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.657227    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc" event={"ID":"63f881f7-c955-411e-8c47-f87e0c94b581","Type":"ContainerDied","Data":"434a32c72757078af05190f975bea569a2d919d6ff5ca2f63cb951e9bf06b023"}
Oct 31 14:01:39.666781 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.666758    4428 scope.go:122] "RemoveContainer" containerID="72c31c199f1ced1b81b30de9b02871d16bd06f3cf4225d41efd2d26897a3a776"
Oct 31 14:01:39.667165 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:01:39.667136    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72c31c199f1ced1b81b30de9b02871d16bd06f3cf4225d41efd2d26897a3a776\": container with ID starting with 72c31c199f1ced1b81b30de9b02871d16bd06f3cf4225d41efd2d26897a3a776 not found: ID does not exist" containerID="72c31c199f1ced1b81b30de9b02871d16bd06f3cf4225d41efd2d26897a3a776"
Oct 31 14:01:39.667252 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.667162    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c31c199f1ced1b81b30de9b02871d16bd06f3cf4225d41efd2d26897a3a776"} err="failed to get container status \"72c31c199f1ced1b81b30de9b02871d16bd06f3cf4225d41efd2d26897a3a776\": rpc error: code = NotFound desc = could not find container \"72c31c199f1ced1b81b30de9b02871d16bd06f3cf4225d41efd2d26897a3a776\": container with ID starting with 72c31c199f1ced1b81b30de9b02871d16bd06f3cf4225d41efd2d26897a3a776 not found: ID does not exist"
Oct 31 14:01:39.667252 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.667182    4428 scope.go:122] "RemoveContainer" containerID="90e8601b0f8153fdc7322c0f79b91ee8c65bb789d22833a17e893d2e0bbd58bd"
Oct 31 14:01:39.677491 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.677470    4428 scope.go:122] "RemoveContainer" containerID="90e8601b0f8153fdc7322c0f79b91ee8c65bb789d22833a17e893d2e0bbd58bd"
Oct 31 14:01:39.677808 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:01:39.677787    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e8601b0f8153fdc7322c0f79b91ee8c65bb789d22833a17e893d2e0bbd58bd\": container with ID starting with 90e8601b0f8153fdc7322c0f79b91ee8c65bb789d22833a17e893d2e0bbd58bd not found: ID does not exist" containerID="90e8601b0f8153fdc7322c0f79b91ee8c65bb789d22833a17e893d2e0bbd58bd"
Oct 31 14:01:39.677904 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.677817    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e8601b0f8153fdc7322c0f79b91ee8c65bb789d22833a17e893d2e0bbd58bd"} err="failed to get container status \"90e8601b0f8153fdc7322c0f79b91ee8c65bb789d22833a17e893d2e0bbd58bd\": rpc error: code = NotFound desc = could not find container \"90e8601b0f8153fdc7322c0f79b91ee8c65bb789d22833a17e893d2e0bbd58bd\": container with ID starting with 90e8601b0f8153fdc7322c0f79b91ee8c65bb789d22833a17e893d2e0bbd58bd not found: ID does not exist"
Oct 31 14:01:39.682383 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.682356    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"]
Oct 31 14:01:39.689129 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.689101    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-56986d4c6c-4knt8"]
Oct 31 14:01:39.701108 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.701085    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"]
Oct 31 14:01:39.705480 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:39.705454    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-586469-7b8b9f445-scvfc"]
Oct 31 14:01:40.201442 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:40.198469    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:40.777301 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:40.777270    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="335e2046-6ac2-4332-a3d3-b721751681df" path="/var/lib/kubelet/pods/335e2046-6ac2-4332-a3d3-b721751681df/volumes"
Oct 31 14:01:40.777738 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:40.777724    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f881f7-c955-411e-8c47-f87e0c94b581" path="/var/lib/kubelet/pods/63f881f7-c955-411e-8c47-f87e0c94b581/volumes"
Oct 31 14:01:41.059955 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:41.059554    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:41.161908 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:41.161291    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"]
Oct 31 14:01:41.282056 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:41.281987    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:41.452541 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:41.452454    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:41.703336 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:41.703069    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb" podUID="35f82843-6eb3-490b-8c6c-9dad7cce21a1" containerName="payload-pre-processing" containerID="cri-o://88943beab6bab49dd7d59d59d160fd7334fab1952768aac0ffa52a5baa3d7ea8" gracePeriod=25
Oct 31 14:01:41.723314 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:41.723282    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:41.899762 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:41.899726    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:42.088153 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.083668    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:42.120893 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.115080    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb" podUID="35f82843-6eb3-490b-8c6c-9dad7cce21a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:01:42.495936 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.488531    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:42.612402 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.612355    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"]
Oct 31 14:01:42.613110 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613090    4428 cpu_manager.go:414] "containerMap: removing container" podUID="c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb" containerName="main"
Oct 31 14:01:42.613175 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613111    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb" containerName="main"
Oct 31 14:01:42.613175 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613120    4428 cpu_manager.go:414] "containerMap: removing container" podUID="2c55141e-2afe-4d0d-ba77-834eca70c27f" containerName="payload-pre-processing"
Oct 31 14:01:42.613175 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613127    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c55141e-2afe-4d0d-ba77-834eca70c27f" containerName="payload-pre-processing"
Oct 31 14:01:42.613175 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613139    4428 cpu_manager.go:414] "containerMap: removing container" podUID="63f881f7-c955-411e-8c47-f87e0c94b581" containerName="payload-processing"
Oct 31 14:01:42.613175 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613144    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f881f7-c955-411e-8c47-f87e0c94b581" containerName="payload-processing"
Oct 31 14:01:42.613175 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613156    4428 cpu_manager.go:414] "containerMap: removing container" podUID="335e2046-6ac2-4332-a3d3-b721751681df" containerName="payload-pre-processing"
Oct 31 14:01:42.613175 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613161    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="335e2046-6ac2-4332-a3d3-b721751681df" containerName="payload-pre-processing"
Oct 31 14:01:42.613175 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613169    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4608de93-17e1-4803-a234-bfb4e24e559f" containerName="istio-proxy"
Oct 31 14:01:42.613175 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613174    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4608de93-17e1-4803-a234-bfb4e24e559f" containerName="istio-proxy"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613183    4428 cpu_manager.go:414] "containerMap: removing container" podUID="71b06a52-1d53-48e7-a5bd-b25f92a89e91" containerName="payload-processing"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613189    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b06a52-1d53-48e7-a5bd-b25f92a89e91" containerName="payload-processing"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613195    4428 cpu_manager.go:414] "containerMap: removing container" podUID="c2da70ae-abad-42d5-be21-0fd2b83848e0" containerName="istio-proxy"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613200    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2da70ae-abad-42d5-be21-0fd2b83848e0" containerName="istio-proxy"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613210    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9825f1d9-236b-4516-94b1-058e389a702c" containerName="payload-pre-processing"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613216    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9825f1d9-236b-4516-94b1-058e389a702c" containerName="payload-pre-processing"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613234    4428 cpu_manager.go:414] "containerMap: removing container" podUID="c2da70ae-abad-42d5-be21-0fd2b83848e0" containerName="istio-proxy"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613239    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2da70ae-abad-42d5-be21-0fd2b83848e0" containerName="istio-proxy"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613246    4428 cpu_manager.go:414] "containerMap: removing container" podUID="66b6b2a9-e6b8-4848-9fb7-4886a7db34e4" containerName="payload-processing"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613250    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b6b2a9-e6b8-4848-9fb7-4886a7db34e4" containerName="payload-processing"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613257    4428 cpu_manager.go:414] "containerMap: removing container" podUID="75240edd-b28b-4986-992b-6557ad0f41be" containerName="maas-api"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613261    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75240edd-b28b-4986-992b-6557ad0f41be" containerName="maas-api"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613386    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6bd76b8-0062-40ce-a682-5ad4ee1c0ccb" containerName="main"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613401    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c55141e-2afe-4d0d-ba77-834eca70c27f" containerName="payload-pre-processing"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613415    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f881f7-c955-411e-8c47-f87e0c94b581" containerName="payload-processing"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613426    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="335e2046-6ac2-4332-a3d3-b721751681df" containerName="payload-pre-processing"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613434    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4608de93-17e1-4803-a234-bfb4e24e559f" containerName="istio-proxy"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613442    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b06a52-1d53-48e7-a5bd-b25f92a89e91" containerName="payload-processing"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613451    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2da70ae-abad-42d5-be21-0fd2b83848e0" containerName="istio-proxy"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613461    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9825f1d9-236b-4516-94b1-058e389a702c" containerName="payload-pre-processing"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613473    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2da70ae-abad-42d5-be21-0fd2b83848e0" containerName="istio-proxy"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613481    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b6b2a9-e6b8-4848-9fb7-4886a7db34e4" containerName="payload-processing"
Oct 31 14:01:42.613500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.613491    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="75240edd-b28b-4986-992b-6557ad0f41be" containerName="maas-api"
Oct 31 14:01:42.619092 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.619065    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"
Oct 31 14:01:42.628119 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.628090    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:01:42.628255 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.628225    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:01:42.632620 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.632559    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"]
Oct 31 14:01:42.683059 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.683020    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:42.719550 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.719511    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66\" (UID: \"14a1e585-d57b-4c90-b9d7-0b7007e3e01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"
Oct 31 14:01:42.719764 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.719560    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fcwk\" (UniqueName: \"kubernetes.io/projected/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-kube-api-access-9fcwk\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66\" (UID: \"14a1e585-d57b-4c90-b9d7-0b7007e3e01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"
Oct 31 14:01:42.719854 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.719830    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66\" (UID: \"14a1e585-d57b-4c90-b9d7-0b7007e3e01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"
Oct 31 14:01:42.821418 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.821315    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66\" (UID: \"14a1e585-d57b-4c90-b9d7-0b7007e3e01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"
Oct 31 14:01:42.821642 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.821464    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66\" (UID: \"14a1e585-d57b-4c90-b9d7-0b7007e3e01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"
Oct 31 14:01:42.821642 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.821505    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fcwk\" (UniqueName: \"kubernetes.io/projected/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-kube-api-access-9fcwk\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66\" (UID: \"14a1e585-d57b-4c90-b9d7-0b7007e3e01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"
Oct 31 14:01:42.822294 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.822265    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66\" (UID: \"14a1e585-d57b-4c90-b9d7-0b7007e3e01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"
Oct 31 14:01:42.824155 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.824131    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66\" (UID: \"14a1e585-d57b-4c90-b9d7-0b7007e3e01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"
Oct 31 14:01:42.834826 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.834783    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fcwk\" (UniqueName: \"kubernetes.io/projected/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-kube-api-access-9fcwk\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66\" (UID: \"14a1e585-d57b-4c90-b9d7-0b7007e3e01c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"
Oct 31 14:01:42.937240 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:42.937202    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"
Oct 31 14:01:43.272819 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.272333    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"]
Oct 31 14:01:43.275551 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:01:43.275513    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a1e585_d57b_4c90_b9d7_0b7007e3e01c.slice/crio-f0ceddb189e8c32317cca6457f2a2a337e7b5d2748df1e5d3d2fac15c33ff241 WatchSource:0}: Error finding container f0ceddb189e8c32317cca6457f2a2a337e7b5d2748df1e5d3d2fac15c33ff241: Status 404 returned error can't find the container with id f0ceddb189e8c32317cca6457f2a2a337e7b5d2748df1e5d3d2fac15c33ff241
Oct 31 14:01:43.318313 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.318179    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:43.686789 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.686761    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"
Oct 31 14:01:43.698949 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.698918    4428 generic.go:356] "Generic (PLEG): container finished" podID="6bfa6cd9-13c7-4201-9a1f-96877ae4288c" containerID="5e133d39a91861a42f5cbeeff03d38c160409a1e61d226e4217d937ee98ae362" exitCode=0
Oct 31 14:01:43.699131 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.698970    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"
Oct 31 14:01:43.699131 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.699004    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs" event={"ID":"6bfa6cd9-13c7-4201-9a1f-96877ae4288c","Type":"ContainerDied","Data":"5e133d39a91861a42f5cbeeff03d38c160409a1e61d226e4217d937ee98ae362"}
Oct 31 14:01:43.699131 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.699112    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs" event={"ID":"6bfa6cd9-13c7-4201-9a1f-96877ae4288c","Type":"ContainerDied","Data":"9a45d56fd6194b74b9766963e1fa8fdc781ac9acd4726913b6a3f55cc7537620"}
Oct 31 14:01:43.699313 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.699158    4428 scope.go:122] "RemoveContainer" containerID="5e133d39a91861a42f5cbeeff03d38c160409a1e61d226e4217d937ee98ae362"
Oct 31 14:01:43.701213 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.701067    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66" event={"ID":"14a1e585-d57b-4c90-b9d7-0b7007e3e01c","Type":"ContainerStarted","Data":"f0ceddb189e8c32317cca6457f2a2a337e7b5d2748df1e5d3d2fac15c33ff241"}
Oct 31 14:01:43.710308 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.710231    4428 scope.go:122] "RemoveContainer" containerID="5e133d39a91861a42f5cbeeff03d38c160409a1e61d226e4217d937ee98ae362"
Oct 31 14:01:43.710575 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:01:43.710540    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e133d39a91861a42f5cbeeff03d38c160409a1e61d226e4217d937ee98ae362\": container with ID starting with 5e133d39a91861a42f5cbeeff03d38c160409a1e61d226e4217d937ee98ae362 not found: ID does not exist" containerID="5e133d39a91861a42f5cbeeff03d38c160409a1e61d226e4217d937ee98ae362"
Oct 31 14:01:43.710680 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.710583    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e133d39a91861a42f5cbeeff03d38c160409a1e61d226e4217d937ee98ae362"} err="failed to get container status \"5e133d39a91861a42f5cbeeff03d38c160409a1e61d226e4217d937ee98ae362\": rpc error: code = NotFound desc = could not find container \"5e133d39a91861a42f5cbeeff03d38c160409a1e61d226e4217d937ee98ae362\": container with ID starting with 5e133d39a91861a42f5cbeeff03d38c160409a1e61d226e4217d937ee98ae362 not found: ID does not exist"
Oct 31 14:01:43.734805 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.734767    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bfa6cd9-13c7-4201-9a1f-96877ae4288c-kube-api-access-l2zr2\" (UniqueName: \"kubernetes.io/projected/6bfa6cd9-13c7-4201-9a1f-96877ae4288c-kube-api-access-l2zr2\") pod \"6bfa6cd9-13c7-4201-9a1f-96877ae4288c\" (UID: \"6bfa6cd9-13c7-4201-9a1f-96877ae4288c\") "
Oct 31 14:01:43.735004 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.734817    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bfa6cd9-13c7-4201-9a1f-96877ae4288c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfa6cd9-13c7-4201-9a1f-96877ae4288c-plugins-config-volume\") pod \"6bfa6cd9-13c7-4201-9a1f-96877ae4288c\" (UID: \"6bfa6cd9-13c7-4201-9a1f-96877ae4288c\") "
Oct 31 14:01:43.735254 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.735229    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bfa6cd9-13c7-4201-9a1f-96877ae4288c-plugins-config-volume" pod "6bfa6cd9-13c7-4201-9a1f-96877ae4288c" (UID: "6bfa6cd9-13c7-4201-9a1f-96877ae4288c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:43.735556 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.735528    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfa6cd9-13c7-4201-9a1f-96877ae4288c-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:43.737283 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.737256    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bfa6cd9-13c7-4201-9a1f-96877ae4288c-kube-api-access-l2zr2" pod "6bfa6cd9-13c7-4201-9a1f-96877ae4288c" (UID: "6bfa6cd9-13c7-4201-9a1f-96877ae4288c"). InnerVolumeSpecName "kube-api-access-l2zr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:43.837387 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.837335    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2zr2\" (UniqueName: \"kubernetes.io/projected/6bfa6cd9-13c7-4201-9a1f-96877ae4288c-kube-api-access-l2zr2\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:43.975973 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:43.975938    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:44.031327 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.031288    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"]
Oct 31 14:01:44.034899 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.034843    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"]
Oct 31 14:01:44.035679 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.035657    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4608de93-17e1-4803-a234-bfb4e24e559f" containerName="istio-proxy"
Oct 31 14:01:44.035811 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.035680    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4608de93-17e1-4803-a234-bfb4e24e559f" containerName="istio-proxy"
Oct 31 14:01:44.035811 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.035695    4428 cpu_manager.go:414] "containerMap: removing container" podUID="6bfa6cd9-13c7-4201-9a1f-96877ae4288c" containerName="payload-processing"
Oct 31 14:01:44.035811 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.035703    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bfa6cd9-13c7-4201-9a1f-96877ae4288c" containerName="payload-processing"
Oct 31 14:01:44.035811 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.035805    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bfa6cd9-13c7-4201-9a1f-96877ae4288c" containerName="payload-processing"
Oct 31 14:01:44.036129 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.035825    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4608de93-17e1-4803-a234-bfb4e24e559f" containerName="istio-proxy"
Oct 31 14:01:44.039695 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.039676    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-974711-5c9c544565-7dvrs"]
Oct 31 14:01:44.039782 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.039770    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"
Oct 31 14:01:44.058948 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.058024    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-586469\""
Oct 31 14:01:44.063891 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.060262    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"]
Oct 31 14:01:44.079085 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.073154    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-586469-dockercfg-2nlsv\""
Oct 31 14:01:44.088973 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.085081    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"]
Oct 31 14:01:44.091089 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.091064    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"
Oct 31 14:01:44.097543 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.097389    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"]
Oct 31 14:01:44.130262 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.130222    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:44.152567 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.150264    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c37feae-b089-45dd-a74e-d52516c48a67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj\" (UID: \"1c37feae-b089-45dd-a74e-d52516c48a67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"
Oct 31 14:01:44.152567 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.150343    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e024838-48ac-410b-8cce-cfb5f6ee6ef9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt\" (UID: \"4e024838-48ac-410b-8cce-cfb5f6ee6ef9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"
Oct 31 14:01:44.152567 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.150405    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw75r\" (UniqueName: \"kubernetes.io/projected/4e024838-48ac-410b-8cce-cfb5f6ee6ef9-kube-api-access-bw75r\") pod \"payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt\" (UID: \"4e024838-48ac-410b-8cce-cfb5f6ee6ef9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"
Oct 31 14:01:44.152567 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.150594    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp2vd\" (UniqueName: \"kubernetes.io/projected/1c37feae-b089-45dd-a74e-d52516c48a67-kube-api-access-mp2vd\") pod \"payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj\" (UID: \"1c37feae-b089-45dd-a74e-d52516c48a67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"
Oct 31 14:01:44.283896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.281644    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e024838-48ac-410b-8cce-cfb5f6ee6ef9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt\" (UID: \"4e024838-48ac-410b-8cce-cfb5f6ee6ef9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"
Oct 31 14:01:44.283896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.281722    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw75r\" (UniqueName: \"kubernetes.io/projected/4e024838-48ac-410b-8cce-cfb5f6ee6ef9-kube-api-access-bw75r\") pod \"payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt\" (UID: \"4e024838-48ac-410b-8cce-cfb5f6ee6ef9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"
Oct 31 14:01:44.283896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.281932    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp2vd\" (UniqueName: \"kubernetes.io/projected/1c37feae-b089-45dd-a74e-d52516c48a67-kube-api-access-mp2vd\") pod \"payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj\" (UID: \"1c37feae-b089-45dd-a74e-d52516c48a67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"
Oct 31 14:01:44.283896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.282039    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c37feae-b089-45dd-a74e-d52516c48a67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj\" (UID: \"1c37feae-b089-45dd-a74e-d52516c48a67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"
Oct 31 14:01:44.306092 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.300027    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e024838-48ac-410b-8cce-cfb5f6ee6ef9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt\" (UID: \"4e024838-48ac-410b-8cce-cfb5f6ee6ef9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"
Oct 31 14:01:44.321069 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.321010    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c37feae-b089-45dd-a74e-d52516c48a67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj\" (UID: \"1c37feae-b089-45dd-a74e-d52516c48a67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"
Oct 31 14:01:44.325596 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.325557    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw75r\" (UniqueName: \"kubernetes.io/projected/4e024838-48ac-410b-8cce-cfb5f6ee6ef9-kube-api-access-bw75r\") pod \"payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt\" (UID: \"4e024838-48ac-410b-8cce-cfb5f6ee6ef9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"
Oct 31 14:01:44.328335 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.328308    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp2vd\" (UniqueName: \"kubernetes.io/projected/1c37feae-b089-45dd-a74e-d52516c48a67-kube-api-access-mp2vd\") pod \"payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj\" (UID: \"1c37feae-b089-45dd-a74e-d52516c48a67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"
Oct 31 14:01:44.387591 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.387503    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"
Oct 31 14:01:44.413227 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.413199    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"
Oct 31 14:01:44.619209 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.619164    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:44.709039 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.708999    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"]
Oct 31 14:01:44.709317 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:01:44.709293    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e024838_48ac_410b_8cce_cfb5f6ee6ef9.slice/crio-022df114e5f77556c0872c3d31f4f5436580378c2d7048f751a712371bdee903 WatchSource:0}: Error finding container 022df114e5f77556c0872c3d31f4f5436580378c2d7048f751a712371bdee903: Status 404 returned error can't find the container with id 022df114e5f77556c0872c3d31f4f5436580378c2d7048f751a712371bdee903
Oct 31 14:01:44.713045 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.713018    4428 generic.go:356] "Generic (PLEG): container finished" podID="14a1e585-d57b-4c90-b9d7-0b7007e3e01c" containerID="8ea423ff03676f42fa3c5aed52140c2e7ef08b9ab5196f8ee180cdfb9a3ecc66" exitCode=0
Oct 31 14:01:44.713137 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.713098    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66" event={"ID":"14a1e585-d57b-4c90-b9d7-0b7007e3e01c","Type":"ContainerDied","Data":"8ea423ff03676f42fa3c5aed52140c2e7ef08b9ab5196f8ee180cdfb9a3ecc66"}
Oct 31 14:01:44.769907 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.768924    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"]
Oct 31 14:01:44.781416 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:44.781384    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bfa6cd9-13c7-4201-9a1f-96877ae4288c" path="/var/lib/kubelet/pods/6bfa6cd9-13c7-4201-9a1f-96877ae4288c/volumes"
Oct 31 14:01:45.222963 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.222923    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"]
Oct 31 14:01:45.223791 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.223224    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58" podUID="31bdda64-b266-4073-b001-e8313083d4fa" containerName="maas-api" containerID="cri-o://d2d92a1a5c8aadeb02847fa8a5588a213215113abd99cc188e4fc7b79fcd78cf" gracePeriod=30
Oct 31 14:01:45.327806 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.327738    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:45.509022 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.507858    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:45.703909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.701467    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:45.726677 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.726484    4428 generic.go:356] "Generic (PLEG): container finished" podID="31bdda64-b266-4073-b001-e8313083d4fa" containerID="d2d92a1a5c8aadeb02847fa8a5588a213215113abd99cc188e4fc7b79fcd78cf" exitCode=0
Oct 31 14:01:45.726677 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.726590    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58" event={"ID":"31bdda64-b266-4073-b001-e8313083d4fa","Type":"ContainerDied","Data":"d2d92a1a5c8aadeb02847fa8a5588a213215113abd99cc188e4fc7b79fcd78cf"}
Oct 31 14:01:45.797483 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.797369    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj" event={"ID":"1c37feae-b089-45dd-a74e-d52516c48a67","Type":"ContainerStarted","Data":"392317b6cb6029d6023b2ecfb6eebe05b0a075fb5f850a352c7866fd33d459f2"}
Oct 31 14:01:45.797483 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.797444    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj" event={"ID":"1c37feae-b089-45dd-a74e-d52516c48a67","Type":"ContainerStarted","Data":"3b7df37e1f7780275f46a71a14fb434028457c4ff87b2e7206483e6b718a9bcf"}
Oct 31 14:01:45.797762 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.797717    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"
Oct 31 14:01:45.804990 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.804909    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt" event={"ID":"4e024838-48ac-410b-8cce-cfb5f6ee6ef9","Type":"ContainerStarted","Data":"b1e840fee4bd4d02a1ecf5f42995f96d34f8a3dee4ad3c3219f4c5a1991198cd"}
Oct 31 14:01:45.804990 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.804968    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt" event={"ID":"4e024838-48ac-410b-8cce-cfb5f6ee6ef9","Type":"ContainerStarted","Data":"022df114e5f77556c0872c3d31f4f5436580378c2d7048f751a712371bdee903"}
Oct 31 14:01:45.805260 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.805150    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"
Oct 31 14:01:45.819965 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.819905    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj" podStartSLOduration=1.819846157 podStartE2EDuration="1.819846157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:01:45.817405308 +0000 UTC m=+3503.215753935" watchObservedRunningTime="2026-10-31 14:01:45.819846157 +0000 UTC m=+3503.218194787"
Oct 31 14:01:45.843399 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.843337    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt" podStartSLOduration=1.8433191880000002 podStartE2EDuration="1.843319188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:01:45.838936525 +0000 UTC m=+3503.237285148" watchObservedRunningTime="2026-10-31 14:01:45.843319188 +0000 UTC m=+3503.241667811"
Oct 31 14:01:45.865913 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.864396    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:45.865913 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.865834    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 14:01:45.907402 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.906092    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31bdda64-b266-4073-b001-e8313083d4fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31bdda64-b266-4073-b001-e8313083d4fa-maas-api-tls\") pod \"31bdda64-b266-4073-b001-e8313083d4fa\" (UID: \"31bdda64-b266-4073-b001-e8313083d4fa\") "
Oct 31 14:01:45.907402 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.906187    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31bdda64-b266-4073-b001-e8313083d4fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bdda64-b266-4073-b001-e8313083d4fa-metrics-certs\") pod \"31bdda64-b266-4073-b001-e8313083d4fa\" (UID: \"31bdda64-b266-4073-b001-e8313083d4fa\") "
Oct 31 14:01:45.907402 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.906284    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31bdda64-b266-4073-b001-e8313083d4fa-kube-api-access-krzkl\" (UniqueName: \"kubernetes.io/projected/31bdda64-b266-4073-b001-e8313083d4fa-kube-api-access-krzkl\") pod \"31bdda64-b266-4073-b001-e8313083d4fa\" (UID: \"31bdda64-b266-4073-b001-e8313083d4fa\") "
Oct 31 14:01:45.907402 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.906393    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31bdda64-b266-4073-b001-e8313083d4fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bdda64-b266-4073-b001-e8313083d4fa-openshift-service-ca\") pod \"31bdda64-b266-4073-b001-e8313083d4fa\" (UID: \"31bdda64-b266-4073-b001-e8313083d4fa\") "
Oct 31 14:01:45.913091 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.913003    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31bdda64-b266-4073-b001-e8313083d4fa-openshift-service-ca" pod "31bdda64-b266-4073-b001-e8313083d4fa" (UID: "31bdda64-b266-4073-b001-e8313083d4fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:45.923134 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.923091    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31bdda64-b266-4073-b001-e8313083d4fa-kube-api-access-krzkl" pod "31bdda64-b266-4073-b001-e8313083d4fa" (UID: "31bdda64-b266-4073-b001-e8313083d4fa"). InnerVolumeSpecName "kube-api-access-krzkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:45.923539 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.923511    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31bdda64-b266-4073-b001-e8313083d4fa-maas-api-tls" pod "31bdda64-b266-4073-b001-e8313083d4fa" (UID: "31bdda64-b266-4073-b001-e8313083d4fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:45.924815 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:45.924783    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31bdda64-b266-4073-b001-e8313083d4fa-metrics-certs" pod "31bdda64-b266-4073-b001-e8313083d4fa" (UID: "31bdda64-b266-4073-b001-e8313083d4fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:46.021895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.014068    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bdda64-b266-4073-b001-e8313083d4fa-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:46.021895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.014117    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31bdda64-b266-4073-b001-e8313083d4fa-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:46.021895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.014131    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bdda64-b266-4073-b001-e8313083d4fa-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:46.021895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.014144    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krzkl\" (UniqueName: \"kubernetes.io/projected/31bdda64-b266-4073-b001-e8313083d4fa-kube-api-access-krzkl\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:46.055168 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.054190    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"
Oct 31 14:01:46.115656 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.115604    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:46.116552 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.116521    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-maas-api-xks-ca\") pod \"14a1e585-d57b-4c90-b9d7-0b7007e3e01c\" (UID: \"14a1e585-d57b-4c90-b9d7-0b7007e3e01c\") "
Oct 31 14:01:46.116685 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.116583    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-maas-api-openshift-ca\") pod \"14a1e585-d57b-4c90-b9d7-0b7007e3e01c\" (UID: \"14a1e585-d57b-4c90-b9d7-0b7007e3e01c\") "
Oct 31 14:01:46.116685 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.116651    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-kube-api-access-9fcwk\" (UniqueName: \"kubernetes.io/projected/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-kube-api-access-9fcwk\") pod \"14a1e585-d57b-4c90-b9d7-0b7007e3e01c\" (UID: \"14a1e585-d57b-4c90-b9d7-0b7007e3e01c\") "
Oct 31 14:01:46.120376 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.120341    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-maas-api-openshift-ca" pod "14a1e585-d57b-4c90-b9d7-0b7007e3e01c" (UID: "14a1e585-d57b-4c90-b9d7-0b7007e3e01c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:46.121803 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.121764    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-maas-api-xks-ca" pod "14a1e585-d57b-4c90-b9d7-0b7007e3e01c" (UID: "14a1e585-d57b-4c90-b9d7-0b7007e3e01c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:46.130769 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.130734    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-kube-api-access-9fcwk" pod "14a1e585-d57b-4c90-b9d7-0b7007e3e01c" (UID: "14a1e585-d57b-4c90-b9d7-0b7007e3e01c"). InnerVolumeSpecName "kube-api-access-9fcwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:46.161798 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.161701    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8" podUID="bf16863d-965a-458c-bf0d-966d50096325" containerName="payload-pre-processing" containerID="cri-o://ced40dcb0459fd8ec4eb8c8770c524bf1b11f317fd7c9878172b5e4e1cad1409" gracePeriod=25
Oct 31 14:01:46.218939 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.218899    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:46.218939 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.218939    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:46.219163 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.218956    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fcwk\" (UniqueName: \"kubernetes.io/projected/14a1e585-d57b-4c90-b9d7-0b7007e3e01c-kube-api-access-9fcwk\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:46.634892 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.633512    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:46.815224 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.815186    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66" event={"ID":"14a1e585-d57b-4c90-b9d7-0b7007e3e01c","Type":"ContainerDied","Data":"f0ceddb189e8c32317cca6457f2a2a337e7b5d2748df1e5d3d2fac15c33ff241"}
Oct 31 14:01:46.815430 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.815229    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"
Oct 31 14:01:46.815430 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.815248    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0ceddb189e8c32317cca6457f2a2a337e7b5d2748df1e5d3d2fac15c33ff241"
Oct 31 14:01:46.816949 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.816921    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58" event={"ID":"31bdda64-b266-4073-b001-e8313083d4fa","Type":"ContainerDied","Data":"49e1963ae566ef887704439c18a246af5cab8a1e3a64fcdd9966775819e1e19d"}
Oct 31 14:01:46.817133 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.816985    4428 scope.go:122] "RemoveContainer" containerID="d2d92a1a5c8aadeb02847fa8a5588a213215113abd99cc188e4fc7b79fcd78cf"
Oct 31 14:01:46.817133 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.816986    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"
Oct 31 14:01:46.842045 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.842009    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"]
Oct 31 14:01:46.849245 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:46.849215    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-33c10a-84db799d5d-wmq58"]
Oct 31 14:01:47.470794 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:47.470736    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:48.001552 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.001519    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"]
Oct 31 14:01:48.002392 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.002370    4428 cpu_manager.go:414] "containerMap: removing container" podUID="31bdda64-b266-4073-b001-e8313083d4fa" containerName="maas-api"
Oct 31 14:01:48.002479 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.002395    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31bdda64-b266-4073-b001-e8313083d4fa" containerName="maas-api"
Oct 31 14:01:48.002479 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.002414    4428 cpu_manager.go:414] "containerMap: removing container" podUID="14a1e585-d57b-4c90-b9d7-0b7007e3e01c" containerName="revoke-keys"
Oct 31 14:01:48.002479 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.002420    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a1e585-d57b-4c90-b9d7-0b7007e3e01c" containerName="revoke-keys"
Oct 31 14:01:48.003528 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.002600    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="31bdda64-b266-4073-b001-e8313083d4fa" containerName="maas-api"
Oct 31 14:01:48.003528 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.002621    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a1e585-d57b-4c90-b9d7-0b7007e3e01c" containerName="revoke-keys"
Oct 31 14:01:48.008303 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.008279    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"
Oct 31 14:01:48.018538 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.018508    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"]
Oct 31 14:01:48.022609 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.022580    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"]
Oct 31 14:01:48.029260 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.029232    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"
Oct 31 14:01:48.043712 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.043671    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db2b73a6-tth5n\" (UID: \"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"
Oct 31 14:01:48.043969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.043945    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b7e18df-36cf-4197-9a90-a2ff9fddc090-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2580d87c-kwqpz\" (UID: \"9b7e18df-36cf-4197-9a90-a2ff9fddc090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"
Oct 31 14:01:48.044170 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.044151    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b7e18df-36cf-4197-9a90-a2ff9fddc090-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2580d87c-kwqpz\" (UID: \"9b7e18df-36cf-4197-9a90-a2ff9fddc090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"
Oct 31 14:01:48.044360 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.044343    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db2b73a6-tth5n\" (UID: \"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"
Oct 31 14:01:48.044523 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.044507    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmnfb\" (UniqueName: \"kubernetes.io/projected/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-kube-api-access-bmnfb\") pod \"maas-api-revoke-sub-db2b73a6-tth5n\" (UID: \"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"
Oct 31 14:01:48.044651 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.044637    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvlb7\" (UniqueName: \"kubernetes.io/projected/9b7e18df-36cf-4197-9a90-a2ff9fddc090-kube-api-access-rvlb7\") pod \"maas-api-revoke-sub-2580d87c-kwqpz\" (UID: \"9b7e18df-36cf-4197-9a90-a2ff9fddc090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"
Oct 31 14:01:48.052447 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.052417    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"]
Oct 31 14:01:48.053977 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.053766    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:01:48.054115 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.053964    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:01:48.146245 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.146206    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db2b73a6-tth5n\" (UID: \"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"
Oct 31 14:01:48.146484 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.146260    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b7e18df-36cf-4197-9a90-a2ff9fddc090-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2580d87c-kwqpz\" (UID: \"9b7e18df-36cf-4197-9a90-a2ff9fddc090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"
Oct 31 14:01:48.146484 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.146316    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b7e18df-36cf-4197-9a90-a2ff9fddc090-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2580d87c-kwqpz\" (UID: \"9b7e18df-36cf-4197-9a90-a2ff9fddc090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"
Oct 31 14:01:48.146484 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.146400    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db2b73a6-tth5n\" (UID: \"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"
Oct 31 14:01:48.146484 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.146458    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmnfb\" (UniqueName: \"kubernetes.io/projected/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-kube-api-access-bmnfb\") pod \"maas-api-revoke-sub-db2b73a6-tth5n\" (UID: \"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"
Oct 31 14:01:48.146721 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.146496    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvlb7\" (UniqueName: \"kubernetes.io/projected/9b7e18df-36cf-4197-9a90-a2ff9fddc090-kube-api-access-rvlb7\") pod \"maas-api-revoke-sub-2580d87c-kwqpz\" (UID: \"9b7e18df-36cf-4197-9a90-a2ff9fddc090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"
Oct 31 14:01:48.147196 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.147166    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b7e18df-36cf-4197-9a90-a2ff9fddc090-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2580d87c-kwqpz\" (UID: \"9b7e18df-36cf-4197-9a90-a2ff9fddc090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"
Oct 31 14:01:48.147448 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.147425    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db2b73a6-tth5n\" (UID: \"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"
Oct 31 14:01:48.149029 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.148998    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db2b73a6-tth5n\" (UID: \"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"
Oct 31 14:01:48.149161 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.149092    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b7e18df-36cf-4197-9a90-a2ff9fddc090-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2580d87c-kwqpz\" (UID: \"9b7e18df-36cf-4197-9a90-a2ff9fddc090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"
Oct 31 14:01:48.159888 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.159843    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvlb7\" (UniqueName: \"kubernetes.io/projected/9b7e18df-36cf-4197-9a90-a2ff9fddc090-kube-api-access-rvlb7\") pod \"maas-api-revoke-sub-2580d87c-kwqpz\" (UID: \"9b7e18df-36cf-4197-9a90-a2ff9fddc090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"
Oct 31 14:01:48.160009 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.159848    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmnfb\" (UniqueName: \"kubernetes.io/projected/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-kube-api-access-bmnfb\") pod \"maas-api-revoke-sub-db2b73a6-tth5n\" (UID: \"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"
Oct 31 14:01:48.333948 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.332962    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"
Oct 31 14:01:48.361919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.359437    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"
Oct 31 14:01:48.472305 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.469669    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:48.694062 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.694027    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"]
Oct 31 14:01:48.694724 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:01:48.694698    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f2ecf3_7c15_4d7f_b33f_75ce662d16c2.slice/crio-b695ce1760bc367804f0bb9eaccddbae22ea534d57279061c6a9110e71013024 WatchSource:0}: Error finding container b695ce1760bc367804f0bb9eaccddbae22ea534d57279061c6a9110e71013024: Status 404 returned error can't find the container with id b695ce1760bc367804f0bb9eaccddbae22ea534d57279061c6a9110e71013024
Oct 31 14:01:48.710752 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.710727    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"]
Oct 31 14:01:48.713407 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:01:48.713376    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7e18df_36cf_4197_9a90_a2ff9fddc090.slice/crio-e87ef1cd12e69e73869ce1b9b86114e2d5742f99064148c07a98ba54dffa7685 WatchSource:0}: Error finding container e87ef1cd12e69e73869ce1b9b86114e2d5742f99064148c07a98ba54dffa7685: Status 404 returned error can't find the container with id e87ef1cd12e69e73869ce1b9b86114e2d5742f99064148c07a98ba54dffa7685
Oct 31 14:01:48.758573 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.758539    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31bdda64-b266-4073-b001-e8313083d4fa" path="/var/lib/kubelet/pods/31bdda64-b266-4073-b001-e8313083d4fa/volumes"
Oct 31 14:01:48.761139 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:01:48.761106    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:01:48.835414 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.835373    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz" event={"ID":"9b7e18df-36cf-4197-9a90-a2ff9fddc090","Type":"ContainerStarted","Data":"e87ef1cd12e69e73869ce1b9b86114e2d5742f99064148c07a98ba54dffa7685"}
Oct 31 14:01:48.837100 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:48.837063    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n" event={"ID":"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2","Type":"ContainerStarted","Data":"b695ce1760bc367804f0bb9eaccddbae22ea534d57279061c6a9110e71013024"}
Oct 31 14:01:49.310308 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:49.310183    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:49.846263 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:49.846217    4428 generic.go:356] "Generic (PLEG): container finished" podID="9b7e18df-36cf-4197-9a90-a2ff9fddc090" containerID="f05921fc7bd367af33f15a33ed942510d44f4d68026ee3fadeeb2b28787c3a6f" exitCode=0
Oct 31 14:01:49.846758 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:49.846730    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz" event={"ID":"9b7e18df-36cf-4197-9a90-a2ff9fddc090","Type":"ContainerDied","Data":"f05921fc7bd367af33f15a33ed942510d44f4d68026ee3fadeeb2b28787c3a6f"}
Oct 31 14:01:49.848636 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:49.848606    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n" event={"ID":"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2","Type":"ContainerStarted","Data":"8d436e33b5ef9763adffba5d0c4e0cde158ec039ae1fe256943316603601168b"}
Oct 31 14:01:49.887964 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:49.887846    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n" podStartSLOduration=2.887821918 podStartE2EDuration="2.887821918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:01:49.886068345 +0000 UTC m=+3507.284416981" watchObservedRunningTime="2026-10-31 14:01:49.887821918 +0000 UTC m=+3507.286170672"
Oct 31 14:01:50.038745 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:50.038648    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:50.830731 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:50.830464    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:50.856630 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:50.856593    4428 generic.go:356] "Generic (PLEG): container finished" podID="f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2" containerID="8d436e33b5ef9763adffba5d0c4e0cde158ec039ae1fe256943316603601168b" exitCode=0
Oct 31 14:01:50.856852 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:50.856677    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n" event={"ID":"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2","Type":"ContainerDied","Data":"8d436e33b5ef9763adffba5d0c4e0cde158ec039ae1fe256943316603601168b"}
Oct 31 14:01:50.858481 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:50.858459    4428 generic.go:356] "Generic (PLEG): container finished" podID="8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" containerID="d57653db8e69ce08b52d338fd62bdce1a79e7e92f409d88643ea1a6dbd738930" exitCode=0
Oct 31 14:01:50.858622 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:50.858544    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m" event={"ID":"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21","Type":"ContainerDied","Data":"d57653db8e69ce08b52d338fd62bdce1a79e7e92f409d88643ea1a6dbd738930"}
Oct 31 14:01:50.859394 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:50.859175    4428 scope.go:122] "RemoveContainer" containerID="d57653db8e69ce08b52d338fd62bdce1a79e7e92f409d88643ea1a6dbd738930"
Oct 31 14:01:50.861256 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:50.861222    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:01:50.861342 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:50.861291    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:01:50.861342 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:50.861325    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:01:50.965564 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:50.965522    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" containerID="cri-o://f3cf6799dbfcfca846aa864ec3c530e9c7aca394c4b8b745f3cf96a04f8a3cc3" gracePeriod=45
Oct 31 14:01:51.100830 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.100802    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"
Oct 31 14:01:51.117052 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.117011    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b7e18df-36cf-4197-9a90-a2ff9fddc090-kube-api-access-rvlb7\" (UniqueName: \"kubernetes.io/projected/9b7e18df-36cf-4197-9a90-a2ff9fddc090-kube-api-access-rvlb7\") pod \"9b7e18df-36cf-4197-9a90-a2ff9fddc090\" (UID: \"9b7e18df-36cf-4197-9a90-a2ff9fddc090\") "
Oct 31 14:01:51.117052 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.117060    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b7e18df-36cf-4197-9a90-a2ff9fddc090-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b7e18df-36cf-4197-9a90-a2ff9fddc090-maas-api-openshift-ca\") pod \"9b7e18df-36cf-4197-9a90-a2ff9fddc090\" (UID: \"9b7e18df-36cf-4197-9a90-a2ff9fddc090\") "
Oct 31 14:01:51.117329 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.117086    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b7e18df-36cf-4197-9a90-a2ff9fddc090-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b7e18df-36cf-4197-9a90-a2ff9fddc090-maas-api-xks-ca\") pod \"9b7e18df-36cf-4197-9a90-a2ff9fddc090\" (UID: \"9b7e18df-36cf-4197-9a90-a2ff9fddc090\") "
Oct 31 14:01:51.117469 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.117445    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b7e18df-36cf-4197-9a90-a2ff9fddc090-maas-api-openshift-ca" pod "9b7e18df-36cf-4197-9a90-a2ff9fddc090" (UID: "9b7e18df-36cf-4197-9a90-a2ff9fddc090"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:51.120069 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.120030    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b7e18df-36cf-4197-9a90-a2ff9fddc090-maas-api-xks-ca" pod "9b7e18df-36cf-4197-9a90-a2ff9fddc090" (UID: "9b7e18df-36cf-4197-9a90-a2ff9fddc090"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:51.120195 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.120152    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7e18df-36cf-4197-9a90-a2ff9fddc090-kube-api-access-rvlb7" pod "9b7e18df-36cf-4197-9a90-a2ff9fddc090" (UID: "9b7e18df-36cf-4197-9a90-a2ff9fddc090"). InnerVolumeSpecName "kube-api-access-rvlb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:51.219138 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.219088    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvlb7\" (UniqueName: \"kubernetes.io/projected/9b7e18df-36cf-4197-9a90-a2ff9fddc090-kube-api-access-rvlb7\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:51.219138 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.219129    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b7e18df-36cf-4197-9a90-a2ff9fddc090-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:51.219138 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.219144    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b7e18df-36cf-4197-9a90-a2ff9fddc090-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:51.308173 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.308142    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 14:01:51.320662 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.320622    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-home\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-home\") pod \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") "
Oct 31 14:01:51.320896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.320756    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-kube-api-access-nvm85\" (UniqueName: \"kubernetes.io/projected/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-kube-api-access-nvm85\") pod \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") "
Oct 31 14:01:51.320896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.320810    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-tmp-dir\") pod \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") "
Oct 31 14:01:51.321050 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.320946    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-dshm\") pod \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") "
Oct 31 14:01:51.321115 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.321037    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-model-cache\") pod \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") "
Oct 31 14:01:51.321271 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.321190    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-tls-certs\" (UniqueName: \"kubernetes.io/secret/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-tls-certs\") pod \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\" (UID: \"0aa9e815-d6f7-4c18-810a-a66a0cb86be6\") "
Oct 31 14:01:51.322079 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.321804    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-model-cache" pod "0aa9e815-d6f7-4c18-810a-a66a0cb86be6" (UID: "0aa9e815-d6f7-4c18-810a-a66a0cb86be6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:51.322079 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.322018    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-tmp-dir" pod "0aa9e815-d6f7-4c18-810a-a66a0cb86be6" (UID: "0aa9e815-d6f7-4c18-810a-a66a0cb86be6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:51.322488 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.322460    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-home" pod "0aa9e815-d6f7-4c18-810a-a66a0cb86be6" (UID: "0aa9e815-d6f7-4c18-810a-a66a0cb86be6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:51.341022 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.340491    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-tls-certs" pod "0aa9e815-d6f7-4c18-810a-a66a0cb86be6" (UID: "0aa9e815-d6f7-4c18-810a-a66a0cb86be6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:51.341480 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.341285    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-dshm" pod "0aa9e815-d6f7-4c18-810a-a66a0cb86be6" (UID: "0aa9e815-d6f7-4c18-810a-a66a0cb86be6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:01:51.344247 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.344204    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-kube-api-access-nvm85" pod "0aa9e815-d6f7-4c18-810a-a66a0cb86be6" (UID: "0aa9e815-d6f7-4c18-810a-a66a0cb86be6"). InnerVolumeSpecName "kube-api-access-nvm85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:51.423070 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.423032    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:51.423070 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.423065    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:51.423070 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.423075    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvm85\" (UniqueName: \"kubernetes.io/projected/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-kube-api-access-nvm85\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:51.423070 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.423085    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:51.423550 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.423094    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:51.423550 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.423102    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aa9e815-d6f7-4c18-810a-a66a0cb86be6-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:51.860773 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.860471    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:51.898450 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.897241    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m" event={"ID":"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21","Type":"ContainerStarted","Data":"556e167c37c8a59ec7c4eea8827eb5c1951251d2acbb5cb92ae5169dcc69bfe8"}
Oct 31 14:01:51.900399 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.900321    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"
Oct 31 14:01:51.900399 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.900357    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz" event={"ID":"9b7e18df-36cf-4197-9a90-a2ff9fddc090","Type":"ContainerDied","Data":"e87ef1cd12e69e73869ce1b9b86114e2d5742f99064148c07a98ba54dffa7685"}
Oct 31 14:01:51.900580 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.900415    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e87ef1cd12e69e73869ce1b9b86114e2d5742f99064148c07a98ba54dffa7685"
Oct 31 14:01:51.901837 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.901814    4428 generic.go:356] "Generic (PLEG): container finished" podID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerID="f3cf6799dbfcfca846aa864ec3c530e9c7aca394c4b8b745f3cf96a04f8a3cc3" exitCode=0
Oct 31 14:01:51.902076 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.902063    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"
Oct 31 14:01:51.903793 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.903556    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" event={"ID":"0aa9e815-d6f7-4c18-810a-a66a0cb86be6","Type":"ContainerDied","Data":"f3cf6799dbfcfca846aa864ec3c530e9c7aca394c4b8b745f3cf96a04f8a3cc3"}
Oct 31 14:01:51.903793 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.903599    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" event={"ID":"0aa9e815-d6f7-4c18-810a-a66a0cb86be6","Type":"ContainerDied","Data":"9cfa3207f47acb660041a3d93ee24486e03cd8c35bc75c542a916adcd11e43d4"}
Oct 31 14:01:51.903793 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.903624    4428 scope.go:122] "RemoveContainer" containerID="f3cf6799dbfcfca846aa864ec3c530e9c7aca394c4b8b745f3cf96a04f8a3cc3"
Oct 31 14:01:51.916498 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.916476    4428 scope.go:122] "RemoveContainer" containerID="f3cf6799dbfcfca846aa864ec3c530e9c7aca394c4b8b745f3cf96a04f8a3cc3"
Oct 31 14:01:51.916813 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:01:51.916791    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3cf6799dbfcfca846aa864ec3c530e9c7aca394c4b8b745f3cf96a04f8a3cc3\": container with ID starting with f3cf6799dbfcfca846aa864ec3c530e9c7aca394c4b8b745f3cf96a04f8a3cc3 not found: ID does not exist" containerID="f3cf6799dbfcfca846aa864ec3c530e9c7aca394c4b8b745f3cf96a04f8a3cc3"
Oct 31 14:01:51.916977 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.916821    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3cf6799dbfcfca846aa864ec3c530e9c7aca394c4b8b745f3cf96a04f8a3cc3"} err="failed to get container status \"f3cf6799dbfcfca846aa864ec3c530e9c7aca394c4b8b745f3cf96a04f8a3cc3\": rpc error: code = NotFound desc = could not find container \"f3cf6799dbfcfca846aa864ec3c530e9c7aca394c4b8b745f3cf96a04f8a3cc3\": container with ID starting with f3cf6799dbfcfca846aa864ec3c530e9c7aca394c4b8b745f3cf96a04f8a3cc3 not found: ID does not exist"
Oct 31 14:01:51.972081 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.971997    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"]
Oct 31 14:01:51.977340 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:51.977290    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln"]
Oct 31 14:01:52.061917 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.042679    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8" podUID="bf16863d-965a-458c-bf0d-966d50096325" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:01:52.096960 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.096924    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb" podUID="35f82843-6eb3-490b-8c6c-9dad7cce21a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:01:52.098586 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.098558    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"
Oct 31 14:01:52.133038 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.132922    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-maas-api-xks-ca\") pod \"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2\" (UID: \"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2\") "
Oct 31 14:01:52.133251 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.133086    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-kube-api-access-bmnfb\" (UniqueName: \"kubernetes.io/projected/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-kube-api-access-bmnfb\") pod \"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2\" (UID: \"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2\") "
Oct 31 14:01:52.133251 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.133153    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-maas-api-openshift-ca\") pod \"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2\" (UID: \"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2\") "
Oct 31 14:01:52.133673 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.133641    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-maas-api-openshift-ca" pod "f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2" (UID: "f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:52.133932 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.133907    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:52.136480 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.135759    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-kube-api-access-bmnfb" pod "f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2" (UID: "f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2"). InnerVolumeSpecName "kube-api-access-bmnfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:52.136964 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.136931    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-maas-api-xks-ca" pod "f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2" (UID: "f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:52.216475 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.216418    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/distinct-w1-d85d82-kserve-74d7d4576c-26mln" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 14:01:52.234761 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.234729    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmnfb\" (UniqueName: \"kubernetes.io/projected/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-kube-api-access-bmnfb\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:52.234761 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.234757    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:52.676789 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.676748    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"]
Oct 31 14:01:52.684884 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.684337    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"]
Oct 31 14:01:52.727495 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.727461    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:52.781807 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.781767    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" path="/var/lib/kubelet/pods/0aa9e815-d6f7-4c18-810a-a66a0cb86be6/volumes"
Oct 31 14:01:52.866130 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.846505    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:01:52.876602 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.876097    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body=
Oct 31 14:01:52.876602 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.876159    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m" podUID="8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused"
Oct 31 14:01:52.886545 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.886516    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:52.911964 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.911931    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"
Oct 31 14:01:52.912172 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.911973    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n" event={"ID":"f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2","Type":"ContainerDied","Data":"b695ce1760bc367804f0bb9eaccddbae22ea534d57279061c6a9110e71013024"}
Oct 31 14:01:52.912172 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:52.912022    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b695ce1760bc367804f0bb9eaccddbae22ea534d57279061c6a9110e71013024"
Oct 31 14:01:53.007264 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:53.007219    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"]
Oct 31 14:01:53.179808 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:53.179767    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:53.360507 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:53.360409    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:53.804600 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:53.804515    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:53.848896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:53.844863    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body=
Oct 31 14:01:53.848896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:53.844985    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m" podUID="8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused"
Oct 31 14:01:53.970892 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:53.970827    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:54.258323 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.258235    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:54.578915 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.578801    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"]
Oct 31 14:01:54.579818 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.579775    4428 cpu_manager.go:414] "containerMap: removing container" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main"
Oct 31 14:01:54.579818 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.579807    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main"
Oct 31 14:01:54.580136 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.579879    4428 cpu_manager.go:414] "containerMap: removing container" podUID="f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2" containerName="revoke-keys"
Oct 31 14:01:54.580136 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.579889    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2" containerName="revoke-keys"
Oct 31 14:01:54.580136 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.579904    4428 cpu_manager.go:414] "containerMap: removing container" podUID="9b7e18df-36cf-4197-9a90-a2ff9fddc090" containerName="revoke-keys"
Oct 31 14:01:54.580136 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.579913    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b7e18df-36cf-4197-9a90-a2ff9fddc090" containerName="revoke-keys"
Oct 31 14:01:54.580136 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.580085    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b7e18df-36cf-4197-9a90-a2ff9fddc090" containerName="revoke-keys"
Oct 31 14:01:54.580136 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.580108    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aa9e815-d6f7-4c18-810a-a66a0cb86be6" containerName="main"
Oct 31 14:01:54.580136 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.580128    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2" containerName="revoke-keys"
Oct 31 14:01:54.584960 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.584935    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"]
Oct 31 14:01:54.585214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.585091    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"
Oct 31 14:01:54.592551 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.590168    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:01:54.592551 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.590239    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:01:54.715315 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.715273    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1a89282-5031-43e8-8a6c-f6d72b79cff1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh\" (UID: \"e1a89282-5031-43e8-8a6c-f6d72b79cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"
Oct 31 14:01:54.715315 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.715319    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1a89282-5031-43e8-8a6c-f6d72b79cff1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh\" (UID: \"e1a89282-5031-43e8-8a6c-f6d72b79cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"
Oct 31 14:01:54.715615 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.715532    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4cn\" (UniqueName: \"kubernetes.io/projected/e1a89282-5031-43e8-8a6c-f6d72b79cff1-kube-api-access-dp4cn\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh\" (UID: \"e1a89282-5031-43e8-8a6c-f6d72b79cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"
Oct 31 14:01:54.816535 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.816497    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1a89282-5031-43e8-8a6c-f6d72b79cff1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh\" (UID: \"e1a89282-5031-43e8-8a6c-f6d72b79cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"
Oct 31 14:01:54.816535 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.816541    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1a89282-5031-43e8-8a6c-f6d72b79cff1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh\" (UID: \"e1a89282-5031-43e8-8a6c-f6d72b79cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"
Oct 31 14:01:54.816830 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.816627    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4cn\" (UniqueName: \"kubernetes.io/projected/e1a89282-5031-43e8-8a6c-f6d72b79cff1-kube-api-access-dp4cn\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh\" (UID: \"e1a89282-5031-43e8-8a6c-f6d72b79cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"
Oct 31 14:01:54.817220 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.817192    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1a89282-5031-43e8-8a6c-f6d72b79cff1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh\" (UID: \"e1a89282-5031-43e8-8a6c-f6d72b79cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"
Oct 31 14:01:54.819293 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.819261    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1a89282-5031-43e8-8a6c-f6d72b79cff1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh\" (UID: \"e1a89282-5031-43e8-8a6c-f6d72b79cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"
Oct 31 14:01:54.830135 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.830063    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4cn\" (UniqueName: \"kubernetes.io/projected/e1a89282-5031-43e8-8a6c-f6d72b79cff1-kube-api-access-dp4cn\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh\" (UID: \"e1a89282-5031-43e8-8a6c-f6d72b79cff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"
Oct 31 14:01:54.844720 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.844690    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body=
Oct 31 14:01:54.844857 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.844738    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m" podUID="8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused"
Oct 31 14:01:54.908093 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:54.908048    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"
Oct 31 14:01:55.081949 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:55.081596    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:55.220360 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:55.220330    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"]
Oct 31 14:01:55.221441 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:01:55.221415    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a89282_5031_43e8_8a6c_f6d72b79cff1.slice/crio-9f454bade534ebdce6318e6b6fc90e5f97d77371143449fa05944c786b9775d4 WatchSource:0}: Error finding container 9f454bade534ebdce6318e6b6fc90e5f97d77371143449fa05944c786b9775d4: Status 404 returned error can't find the container with id 9f454bade534ebdce6318e6b6fc90e5f97d77371143449fa05944c786b9775d4
Oct 31 14:01:55.844774 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:55.844688    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body=
Oct 31 14:01:55.844774 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:55.844759    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m" podUID="8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused"
Oct 31 14:01:55.935359 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:55.935321    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh" event={"ID":"e1a89282-5031-43e8-8a6c-f6d72b79cff1","Type":"ContainerStarted","Data":"ac46682da3634950d34eab0960508be33f5a3f577597626f2007ed1fd66bf4ea"}
Oct 31 14:01:55.935530 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:55.935384    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh" event={"ID":"e1a89282-5031-43e8-8a6c-f6d72b79cff1","Type":"ContainerStarted","Data":"9f454bade534ebdce6318e6b6fc90e5f97d77371143449fa05944c786b9775d4"}
Oct 31 14:01:55.971948 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:55.971845    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh" podStartSLOduration=1.971822839 podStartE2EDuration="1.971822839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:01:55.971235906 +0000 UTC m=+3513.369584530" watchObservedRunningTime="2026-10-31 14:01:55.971822839 +0000 UTC m=+3513.370171467"
Oct 31 14:01:56.307950 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:56.307909    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:56.820049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:56.820017    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"
Oct 31 14:01:56.820275 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:56.820093    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"
Oct 31 14:01:56.844345 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:56.844304    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body=
Oct 31 14:01:56.844552 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:56.844375    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m" podUID="8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused"
Oct 31 14:01:56.968280 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:56.968235    4428 generic.go:356] "Generic (PLEG): container finished" podID="e1a89282-5031-43e8-8a6c-f6d72b79cff1" containerID="ac46682da3634950d34eab0960508be33f5a3f577597626f2007ed1fd66bf4ea" exitCode=0
Oct 31 14:01:56.968471 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:56.968307    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh" event={"ID":"e1a89282-5031-43e8-8a6c-f6d72b79cff1","Type":"ContainerDied","Data":"ac46682da3634950d34eab0960508be33f5a3f577597626f2007ed1fd66bf4ea"}
Oct 31 14:01:57.094126 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:57.094020    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:57.100916 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:57.100876    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:01:57.680484 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:57.680424    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8" podUID="461e6055-9080-4aab-8f66-43139b5b9cc4" containerName="payload-processing" containerID="cri-o://7b681ae63c8fd118ac450541b1dfce433edb2f63b8f9a076af9dcf0bab675d5a" gracePeriod=25
Oct 31 14:01:57.684969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:57.684919    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f" podUID="bab754b1-4f25-4274-a326-579b5326b39a" containerName="payload-pre-processing" containerID="cri-o://30d972a1ed8bc132c0ed4a9aa8ce5e45564c4be675592c69bc7fff839b243452" gracePeriod=25
Oct 31 14:01:57.849151 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:57.849116    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:01:57.914185 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:57.914120    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:57.976508 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:57.976483    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:01:58.007993 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.007947    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v" podUID="2cba5737-a6a1-4c1c-8b1d-24f1f7162a60" containerName="payload-pre-processing" containerID="cri-o://41b23ae38f5092c43300be2a3b088fb8d20417efce971f7d1772bd09bbcb3219" gracePeriod=25
Oct 31 14:01:58.180001 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.179970    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"
Oct 31 14:01:58.305859 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.305742    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1a89282-5031-43e8-8a6c-f6d72b79cff1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1a89282-5031-43e8-8a6c-f6d72b79cff1-maas-api-xks-ca\") pod \"e1a89282-5031-43e8-8a6c-f6d72b79cff1\" (UID: \"e1a89282-5031-43e8-8a6c-f6d72b79cff1\") "
Oct 31 14:01:58.305859 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.305841    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a89282-5031-43e8-8a6c-f6d72b79cff1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1a89282-5031-43e8-8a6c-f6d72b79cff1-maas-api-openshift-ca\") pod \"e1a89282-5031-43e8-8a6c-f6d72b79cff1\" (UID: \"e1a89282-5031-43e8-8a6c-f6d72b79cff1\") "
Oct 31 14:01:58.306182 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.305973    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a89282-5031-43e8-8a6c-f6d72b79cff1-kube-api-access-dp4cn\" (UniqueName: \"kubernetes.io/projected/e1a89282-5031-43e8-8a6c-f6d72b79cff1-kube-api-access-dp4cn\") pod \"e1a89282-5031-43e8-8a6c-f6d72b79cff1\" (UID: \"e1a89282-5031-43e8-8a6c-f6d72b79cff1\") "
Oct 31 14:01:58.306252 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.306219    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a89282-5031-43e8-8a6c-f6d72b79cff1-maas-api-openshift-ca" pod "e1a89282-5031-43e8-8a6c-f6d72b79cff1" (UID: "e1a89282-5031-43e8-8a6c-f6d72b79cff1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:01:58.306478 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.306456    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1a89282-5031-43e8-8a6c-f6d72b79cff1-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:58.308317 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.308291    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1a89282-5031-43e8-8a6c-f6d72b79cff1-maas-api-xks-ca" pod "e1a89282-5031-43e8-8a6c-f6d72b79cff1" (UID: "e1a89282-5031-43e8-8a6c-f6d72b79cff1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:01:58.308422 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.308326    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a89282-5031-43e8-8a6c-f6d72b79cff1-kube-api-access-dp4cn" pod "e1a89282-5031-43e8-8a6c-f6d72b79cff1" (UID: "e1a89282-5031-43e8-8a6c-f6d72b79cff1"). InnerVolumeSpecName "kube-api-access-dp4cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:01:58.407604 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.407565    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp4cn\" (UniqueName: \"kubernetes.io/projected/e1a89282-5031-43e8-8a6c-f6d72b79cff1-kube-api-access-dp4cn\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:58.407604 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.407596    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1a89282-5031-43e8-8a6c-f6d72b79cff1-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:01:58.650104 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.650005    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:58.800120 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.800080    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8" podUID="461e6055-9080-4aab-8f66-43139b5b9cc4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:01:58.813486 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.813455    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:58.819023 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.818992    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"]
Oct 31 14:01:58.822324 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.822303    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-14ba5a-bfc66"]
Oct 31 14:01:58.841896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.841825    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f" podUID="bab754b1-4f25-4274-a326-579b5326b39a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:01:59.005892 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:58.998706    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:59.031612 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:59.029229    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"
Oct 31 14:01:59.031612 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:59.029351    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh" event={"ID":"e1a89282-5031-43e8-8a6c-f6d72b79cff1","Type":"ContainerDied","Data":"9f454bade534ebdce6318e6b6fc90e5f97d77371143449fa05944c786b9775d4"}
Oct 31 14:01:59.031612 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:59.029389    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f454bade534ebdce6318e6b6fc90e5f97d77371143449fa05944c786b9775d4"
Oct 31 14:01:59.195948 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:59.195814    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:59.362665 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:59.362567    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:59.594347 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:59.594312    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:01:59.866771 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:01:59.866730    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:00.217809 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:00.217774    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v" podUID="2cba5737-a6a1-4c1c-8b1d-24f1f7162a60" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:00.315915 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:00.312632    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:00.758317 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:00.758275    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a1e585-d57b-4c90-b9d7-0b7007e3e01c" path="/var/lib/kubelet/pods/14a1e585-d57b-4c90-b9d7-0b7007e3e01c/volumes"
Oct 31 14:02:00.886150 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:00.886117    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:01.120895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:01.120347    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:01.389954 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:01.375458    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:01.567813 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:01.567775    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:01.979663 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:01.979487    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"
Oct 31 14:02:02.042820 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.042764    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8" podUID="bf16863d-965a-458c-bf0d-966d50096325" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:02.061679 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.061629    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f82843-6eb3-490b-8c6c-9dad7cce21a1-kube-api-access-kkv59\" (UniqueName: \"kubernetes.io/projected/35f82843-6eb3-490b-8c6c-9dad7cce21a1-kube-api-access-kkv59\") pod \"35f82843-6eb3-490b-8c6c-9dad7cce21a1\" (UID: \"35f82843-6eb3-490b-8c6c-9dad7cce21a1\") "
Oct 31 14:02:02.061942 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.061780    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f82843-6eb3-490b-8c6c-9dad7cce21a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f82843-6eb3-490b-8c6c-9dad7cce21a1-plugins-config-volume\") pod \"35f82843-6eb3-490b-8c6c-9dad7cce21a1\" (UID: \"35f82843-6eb3-490b-8c6c-9dad7cce21a1\") "
Oct 31 14:02:02.063352 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.063007    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f82843-6eb3-490b-8c6c-9dad7cce21a1-plugins-config-volume" pod "35f82843-6eb3-490b-8c6c-9dad7cce21a1" (UID: "35f82843-6eb3-490b-8c6c-9dad7cce21a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:02.070509 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.070461    4428 generic.go:356] "Generic (PLEG): container finished" podID="35f82843-6eb3-490b-8c6c-9dad7cce21a1" containerID="88943beab6bab49dd7d59d59d160fd7334fab1952768aac0ffa52a5baa3d7ea8" exitCode=0
Oct 31 14:02:02.070682 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.070649    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb" event={"ID":"35f82843-6eb3-490b-8c6c-9dad7cce21a1","Type":"ContainerDied","Data":"88943beab6bab49dd7d59d59d160fd7334fab1952768aac0ffa52a5baa3d7ea8"}
Oct 31 14:02:02.070778 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.070760    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb" event={"ID":"35f82843-6eb3-490b-8c6c-9dad7cce21a1","Type":"ContainerDied","Data":"5932f5e768c4e9223895356f1b100fc182287a6e1f1d3a05c7dd136a8403eb78"}
Oct 31 14:02:02.070841 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.070812    4428 scope.go:122] "RemoveContainer" containerID="88943beab6bab49dd7d59d59d160fd7334fab1952768aac0ffa52a5baa3d7ea8"
Oct 31 14:02:02.071108 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.071089    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"
Oct 31 14:02:02.076658 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.076626    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f82843-6eb3-490b-8c6c-9dad7cce21a1-kube-api-access-kkv59" pod "35f82843-6eb3-490b-8c6c-9dad7cce21a1" (UID: "35f82843-6eb3-490b-8c6c-9dad7cce21a1"). InnerVolumeSpecName "kube-api-access-kkv59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:02.090067 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.090040    4428 scope.go:122] "RemoveContainer" containerID="88943beab6bab49dd7d59d59d160fd7334fab1952768aac0ffa52a5baa3d7ea8"
Oct 31 14:02:02.090441 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:02.090421    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88943beab6bab49dd7d59d59d160fd7334fab1952768aac0ffa52a5baa3d7ea8\": container with ID starting with 88943beab6bab49dd7d59d59d160fd7334fab1952768aac0ffa52a5baa3d7ea8 not found: ID does not exist" containerID="88943beab6bab49dd7d59d59d160fd7334fab1952768aac0ffa52a5baa3d7ea8"
Oct 31 14:02:02.090482 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.090453    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88943beab6bab49dd7d59d59d160fd7334fab1952768aac0ffa52a5baa3d7ea8"} err="failed to get container status \"88943beab6bab49dd7d59d59d160fd7334fab1952768aac0ffa52a5baa3d7ea8\": rpc error: code = NotFound desc = could not find container \"88943beab6bab49dd7d59d59d160fd7334fab1952768aac0ffa52a5baa3d7ea8\": container with ID starting with 88943beab6bab49dd7d59d59d160fd7334fab1952768aac0ffa52a5baa3d7ea8 not found: ID does not exist"
Oct 31 14:02:02.163775 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.163740    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkv59\" (UniqueName: \"kubernetes.io/projected/35f82843-6eb3-490b-8c6c-9dad7cce21a1-kube-api-access-kkv59\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:02.163775 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.163774    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f82843-6eb3-490b-8c6c-9dad7cce21a1-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:02.325950 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.325350    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:02.403937 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.403765    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"]
Oct 31 14:02:02.409121 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.409081    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-33c10a-67fc79744c-pfszb"]
Oct 31 14:02:02.756433 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:02.756394    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f82843-6eb3-490b-8c6c-9dad7cce21a1" path="/var/lib/kubelet/pods/35f82843-6eb3-490b-8c6c-9dad7cce21a1/volumes"
Oct 31 14:02:03.199334 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:03.199238    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:04.110606 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:04.110572    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:04.545788 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:04.545756    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:04.650448 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:04.650399    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"]
Oct 31 14:02:04.711184 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:04.711150    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:04.909359 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:04.909279    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:05.155023 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:05.153736    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:05.802820 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:05.802784    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:06.193477 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.193442    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:06.196583 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.196555    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"]
Oct 31 14:02:06.197308 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.197292    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e1a89282-5031-43e8-8a6c-f6d72b79cff1" containerName="revoke-keys"
Oct 31 14:02:06.197366 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.197310    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a89282-5031-43e8-8a6c-f6d72b79cff1" containerName="revoke-keys"
Oct 31 14:02:06.197366 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.197347    4428 cpu_manager.go:414] "containerMap: removing container" podUID="35f82843-6eb3-490b-8c6c-9dad7cce21a1" containerName="payload-pre-processing"
Oct 31 14:02:06.197366 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.197351    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f82843-6eb3-490b-8c6c-9dad7cce21a1" containerName="payload-pre-processing"
Oct 31 14:02:06.197469 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.197448    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a89282-5031-43e8-8a6c-f6d72b79cff1" containerName="revoke-keys"
Oct 31 14:02:06.197469 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.197462    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f82843-6eb3-490b-8c6c-9dad7cce21a1" containerName="payload-pre-processing"
Oct 31 14:02:06.201041 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.201015    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"
Oct 31 14:02:06.212915 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.212886    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"]
Oct 31 14:02:06.215811 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.215748    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1b75f9\""
Oct 31 14:02:06.216090 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.215927    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1b75f9-dockercfg-6drrk\""
Oct 31 14:02:06.250356 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.250328    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a329323-cfeb-4213-950b-1ce6c8cc3fbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh\" (UID: \"3a329323-cfeb-4213-950b-1ce6c8cc3fbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"
Oct 31 14:02:06.250555 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.250461    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwrq7\" (UniqueName: \"kubernetes.io/projected/3a329323-cfeb-4213-950b-1ce6c8cc3fbe-kube-api-access-xwrq7\") pod \"payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh\" (UID: \"3a329323-cfeb-4213-950b-1ce6c8cc3fbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"
Oct 31 14:02:06.254036 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.254006    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"]
Oct 31 14:02:06.259466 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.259447    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"
Oct 31 14:02:06.274696 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.274668    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"]
Oct 31 14:02:06.405942 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.363554    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22gj8\" (UniqueName: \"kubernetes.io/projected/67e09833-8284-4dfa-a8f8-8f371ec49284-kube-api-access-22gj8\") pod \"payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84\" (UID: \"67e09833-8284-4dfa-a8f8-8f371ec49284\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"
Oct 31 14:02:06.405942 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.363682    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e09833-8284-4dfa-a8f8-8f371ec49284-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84\" (UID: \"67e09833-8284-4dfa-a8f8-8f371ec49284\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"
Oct 31 14:02:06.405942 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.363774    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a329323-cfeb-4213-950b-1ce6c8cc3fbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh\" (UID: \"3a329323-cfeb-4213-950b-1ce6c8cc3fbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"
Oct 31 14:02:06.405942 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.363914    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwrq7\" (UniqueName: \"kubernetes.io/projected/3a329323-cfeb-4213-950b-1ce6c8cc3fbe-kube-api-access-xwrq7\") pod \"payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh\" (UID: \"3a329323-cfeb-4213-950b-1ce6c8cc3fbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"
Oct 31 14:02:06.405942 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.365342    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a329323-cfeb-4213-950b-1ce6c8cc3fbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh\" (UID: \"3a329323-cfeb-4213-950b-1ce6c8cc3fbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"
Oct 31 14:02:06.405942 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.382258    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwrq7\" (UniqueName: \"kubernetes.io/projected/3a329323-cfeb-4213-950b-1ce6c8cc3fbe-kube-api-access-xwrq7\") pod \"payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh\" (UID: \"3a329323-cfeb-4213-950b-1ce6c8cc3fbe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"
Oct 31 14:02:06.405942 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.383643    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:06.466332 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.465979    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22gj8\" (UniqueName: \"kubernetes.io/projected/67e09833-8284-4dfa-a8f8-8f371ec49284-kube-api-access-22gj8\") pod \"payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84\" (UID: \"67e09833-8284-4dfa-a8f8-8f371ec49284\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"
Oct 31 14:02:06.466332 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.466100    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e09833-8284-4dfa-a8f8-8f371ec49284-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84\" (UID: \"67e09833-8284-4dfa-a8f8-8f371ec49284\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"
Oct 31 14:02:06.467151 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.467085    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e09833-8284-4dfa-a8f8-8f371ec49284-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84\" (UID: \"67e09833-8284-4dfa-a8f8-8f371ec49284\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"
Oct 31 14:02:06.481894 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.481721    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22gj8\" (UniqueName: \"kubernetes.io/projected/67e09833-8284-4dfa-a8f8-8f371ec49284-kube-api-access-22gj8\") pod \"payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84\" (UID: \"67e09833-8284-4dfa-a8f8-8f371ec49284\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"
Oct 31 14:02:06.554717 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.554688    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"
Oct 31 14:02:06.588983 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.575188    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"
Oct 31 14:02:06.618006 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.617979    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"
Oct 31 14:02:06.667688 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.667651    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf16863d-965a-458c-bf0d-966d50096325-kube-api-access-96wjp\" (UniqueName: \"kubernetes.io/projected/bf16863d-965a-458c-bf0d-966d50096325-kube-api-access-96wjp\") pod \"bf16863d-965a-458c-bf0d-966d50096325\" (UID: \"bf16863d-965a-458c-bf0d-966d50096325\") "
Oct 31 14:02:06.667849 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.667813    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf16863d-965a-458c-bf0d-966d50096325-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf16863d-965a-458c-bf0d-966d50096325-plugins-config-volume\") pod \"bf16863d-965a-458c-bf0d-966d50096325\" (UID: \"bf16863d-965a-458c-bf0d-966d50096325\") "
Oct 31 14:02:06.669494 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.669111    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf16863d-965a-458c-bf0d-966d50096325-plugins-config-volume" pod "bf16863d-965a-458c-bf0d-966d50096325" (UID: "bf16863d-965a-458c-bf0d-966d50096325"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:06.678106 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.678070    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf16863d-965a-458c-bf0d-966d50096325-kube-api-access-96wjp" pod "bf16863d-965a-458c-bf0d-966d50096325" (UID: "bf16863d-965a-458c-bf0d-966d50096325"). InnerVolumeSpecName "kube-api-access-96wjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:06.769857 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.769825    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96wjp\" (UniqueName: \"kubernetes.io/projected/bf16863d-965a-458c-bf0d-966d50096325-kube-api-access-96wjp\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:06.769857 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.769860    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf16863d-965a-458c-bf0d-966d50096325-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:06.883733 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:02:06.880014    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e09833_8284_4dfa_a8f8_8f371ec49284.slice/crio-9a57fcec7d16e5ab2b9c68f8fbca45a747922bcc62c278034425a5f251fc8a00 WatchSource:0}: Error finding container 9a57fcec7d16e5ab2b9c68f8fbca45a747922bcc62c278034425a5f251fc8a00: Status 404 returned error can't find the container with id 9a57fcec7d16e5ab2b9c68f8fbca45a747922bcc62c278034425a5f251fc8a00
Oct 31 14:02:06.911176 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.911136    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:06.911385 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.911250    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"]
Oct 31 14:02:06.933739 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:06.933709    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"]
Oct 31 14:02:07.109525 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.109425    4428 generic.go:356] "Generic (PLEG): container finished" podID="bf16863d-965a-458c-bf0d-966d50096325" containerID="ced40dcb0459fd8ec4eb8c8770c524bf1b11f317fd7c9878172b5e4e1cad1409" exitCode=0
Oct 31 14:02:07.109525 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.109496    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"
Oct 31 14:02:07.109525 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.109503    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8" event={"ID":"bf16863d-965a-458c-bf0d-966d50096325","Type":"ContainerDied","Data":"ced40dcb0459fd8ec4eb8c8770c524bf1b11f317fd7c9878172b5e4e1cad1409"}
Oct 31 14:02:07.109862 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.109554    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8" event={"ID":"bf16863d-965a-458c-bf0d-966d50096325","Type":"ContainerDied","Data":"87a8ff8cb16203b02fd4358bf3466dfed1ef82953458da70b0b7a58d1cc9fc72"}
Oct 31 14:02:07.109862 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.109580    4428 scope.go:122] "RemoveContainer" containerID="ced40dcb0459fd8ec4eb8c8770c524bf1b11f317fd7c9878172b5e4e1cad1409"
Oct 31 14:02:07.112146 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.111582    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh" event={"ID":"3a329323-cfeb-4213-950b-1ce6c8cc3fbe","Type":"ContainerStarted","Data":"fb941601b93953c2ce81404a15c1ec3c2625aaf4428252aa005a22c95ff33683"}
Oct 31 14:02:07.112146 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.111712    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh" event={"ID":"3a329323-cfeb-4213-950b-1ce6c8cc3fbe","Type":"ContainerStarted","Data":"daf6ca5f1eab4e03f3fab808a0d30fa7f5cfa0fb07da8f6b7f6a6f659ba51e1c"}
Oct 31 14:02:07.112146 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.111746    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"
Oct 31 14:02:07.113785 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.113760    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84" event={"ID":"67e09833-8284-4dfa-a8f8-8f371ec49284","Type":"ContainerStarted","Data":"5dedcfc490f11661731e45a0831b67e1590b18090021042063a468a8e8165a62"}
Oct 31 14:02:07.113937 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.113798    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84" event={"ID":"67e09833-8284-4dfa-a8f8-8f371ec49284","Type":"ContainerStarted","Data":"9a57fcec7d16e5ab2b9c68f8fbca45a747922bcc62c278034425a5f251fc8a00"}
Oct 31 14:02:07.113937 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.113843    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"
Oct 31 14:02:07.120425 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.120404    4428 scope.go:122] "RemoveContainer" containerID="ced40dcb0459fd8ec4eb8c8770c524bf1b11f317fd7c9878172b5e4e1cad1409"
Oct 31 14:02:07.120704 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:07.120684    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced40dcb0459fd8ec4eb8c8770c524bf1b11f317fd7c9878172b5e4e1cad1409\": container with ID starting with ced40dcb0459fd8ec4eb8c8770c524bf1b11f317fd7c9878172b5e4e1cad1409 not found: ID does not exist" containerID="ced40dcb0459fd8ec4eb8c8770c524bf1b11f317fd7c9878172b5e4e1cad1409"
Oct 31 14:02:07.120756 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.120712    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced40dcb0459fd8ec4eb8c8770c524bf1b11f317fd7c9878172b5e4e1cad1409"} err="failed to get container status \"ced40dcb0459fd8ec4eb8c8770c524bf1b11f317fd7c9878172b5e4e1cad1409\": rpc error: code = NotFound desc = could not find container \"ced40dcb0459fd8ec4eb8c8770c524bf1b11f317fd7c9878172b5e4e1cad1409\": container with ID starting with ced40dcb0459fd8ec4eb8c8770c524bf1b11f317fd7c9878172b5e4e1cad1409 not found: ID does not exist"
Oct 31 14:02:07.134431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.134405    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"]
Oct 31 14:02:07.138846 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.138820    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d85d82-678fdc7bff-gbnv8"]
Oct 31 14:02:07.157632 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.157583    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh" podStartSLOduration=1.157568509 podStartE2EDuration="1.157568509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:07.155131223 +0000 UTC m=+3524.553479871" watchObservedRunningTime="2026-10-31 14:02:07.157568509 +0000 UTC m=+3524.555917132"
Oct 31 14:02:07.177680 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.177626    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84" podStartSLOduration=1.177611173 podStartE2EDuration="1.177611173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:07.174140278 +0000 UTC m=+3524.572488903" watchObservedRunningTime="2026-10-31 14:02:07.177611173 +0000 UTC m=+3524.575959797"
Oct 31 14:02:07.942101 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:07.942068    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:08.215281 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:08.215244    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:08.383776 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:08.383746    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:08.484303 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:08.484182    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"]
Oct 31 14:02:08.484496 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:08.484414    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2" podUID="c1dfb224-4987-4a65-9cd3-56a3383aac7f" containerName="istio-proxy" containerID="cri-o://8cc88ea0301d428bd54c91620f0c7e1e87b92176c21a9d8de5e9ef02862128e3" gracePeriod=30
Oct 31 14:02:08.589668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:08.589612    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:08.756252 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:08.756153    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf16863d-965a-458c-bf0d-966d50096325" path="/var/lib/kubelet/pods/bf16863d-965a-458c-bf0d-966d50096325/volumes"
Oct 31 14:02:08.800514 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:08.800466    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8" podUID="461e6055-9080-4aab-8f66-43139b5b9cc4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:08.841178 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:08.841105    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f" podUID="bab754b1-4f25-4274-a326-579b5326b39a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:08.923292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:08.923253    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:09.104590 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.104505    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:09.350900 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.349805    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:09.650892 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.650824    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5" podUID="e8641866-a4d3-4c0d-8d8a-dd25e8572139" containerName="payload-processing" containerID="cri-o://0714db78f0b3bef9c995c8f3310afdf593508b279d76303f783d5246189760d5" gracePeriod=25
Oct 31 14:02:09.655727 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.655696    4428 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:02:09.655852 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.655750    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2" podUID="c1dfb224-4987-4a65-9cd3-56a3383aac7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:02:09.770439 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.770360    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"]
Oct 31 14:02:09.771722 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.771676    4428 cpu_manager.go:414] "containerMap: removing container" podUID="bf16863d-965a-458c-bf0d-966d50096325" containerName="payload-pre-processing"
Oct 31 14:02:09.771722 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.771726    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf16863d-965a-458c-bf0d-966d50096325" containerName="payload-pre-processing"
Oct 31 14:02:09.772089 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.771950    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf16863d-965a-458c-bf0d-966d50096325" containerName="payload-pre-processing"
Oct 31 14:02:09.777260 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.777228    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"
Oct 31 14:02:09.782021 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.781992    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:02:09.782181 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.782060    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:02:09.785094 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.785071    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"]
Oct 31 14:02:09.836407 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.836358    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8922cd48-13c8-4172-9470-736ac36a46ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf\" (UID: \"8922cd48-13c8-4172-9470-736ac36a46ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"
Oct 31 14:02:09.836604 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.836541    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8922cd48-13c8-4172-9470-736ac36a46ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf\" (UID: \"8922cd48-13c8-4172-9470-736ac36a46ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"
Oct 31 14:02:09.836856 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.836824    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcthq\" (UniqueName: \"kubernetes.io/projected/8922cd48-13c8-4172-9470-736ac36a46ed-kube-api-access-xcthq\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf\" (UID: \"8922cd48-13c8-4172-9470-736ac36a46ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"
Oct 31 14:02:09.938302 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.938211    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8922cd48-13c8-4172-9470-736ac36a46ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf\" (UID: \"8922cd48-13c8-4172-9470-736ac36a46ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"
Oct 31 14:02:09.938302 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.938292    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8922cd48-13c8-4172-9470-736ac36a46ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf\" (UID: \"8922cd48-13c8-4172-9470-736ac36a46ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"
Oct 31 14:02:09.938546 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.938421    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcthq\" (UniqueName: \"kubernetes.io/projected/8922cd48-13c8-4172-9470-736ac36a46ed-kube-api-access-xcthq\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf\" (UID: \"8922cd48-13c8-4172-9470-736ac36a46ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"
Oct 31 14:02:09.939053 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.939028    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8922cd48-13c8-4172-9470-736ac36a46ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf\" (UID: \"8922cd48-13c8-4172-9470-736ac36a46ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"
Oct 31 14:02:09.940751 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.940727    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8922cd48-13c8-4172-9470-736ac36a46ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf\" (UID: \"8922cd48-13c8-4172-9470-736ac36a46ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"
Oct 31 14:02:09.952316 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:09.952292    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcthq\" (UniqueName: \"kubernetes.io/projected/8922cd48-13c8-4172-9470-736ac36a46ed-kube-api-access-xcthq\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf\" (UID: \"8922cd48-13c8-4172-9470-736ac36a46ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"
Oct 31 14:02:10.095656 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:10.095624    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"
Oct 31 14:02:10.160456 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:10.160192    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:10.217802 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:10.217768    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v" podUID="2cba5737-a6a1-4c1c-8b1d-24f1f7162a60" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:10.416484 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:10.416449    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"]
Oct 31 14:02:10.973953 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:10.973916    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:11.202945 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:11.192373    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf" event={"ID":"8922cd48-13c8-4172-9470-736ac36a46ed","Type":"ContainerStarted","Data":"eb4721a55ea2f0ddbeae07855741ace0ddca3a2b11d998478e8bb16c873e3d44"}
Oct 31 14:02:11.202945 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:11.192442    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf" event={"ID":"8922cd48-13c8-4172-9470-736ac36a46ed","Type":"ContainerStarted","Data":"f2a0994da8b215d95286b258fc4825854b07c6458b2c32a2be9dbadee6432f88"}
Oct 31 14:02:11.241988 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:11.239547    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:11.244612 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:11.244544    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf" podStartSLOduration=2.2445241830000002 podStartE2EDuration="2.244524183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:11.225457489 +0000 UTC m=+3528.623806130" watchObservedRunningTime="2026-10-31 14:02:11.244524183 +0000 UTC m=+3528.642872808"
Oct 31 14:02:11.344359 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:11.343952    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"]
Oct 31 14:02:11.379893 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:11.372902    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-abdcb1-nb7mh"]
Oct 31 14:02:11.398146 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:11.398117    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:11.558329 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:11.556344    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:11.823373 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:11.823268    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:12.202354 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:12.202257    4428 generic.go:356] "Generic (PLEG): container finished" podID="8922cd48-13c8-4172-9470-736ac36a46ed" containerID="eb4721a55ea2f0ddbeae07855741ace0ddca3a2b11d998478e8bb16c873e3d44" exitCode=0
Oct 31 14:02:12.202354 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:12.202304    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf" event={"ID":"8922cd48-13c8-4172-9470-736ac36a46ed","Type":"ContainerDied","Data":"eb4721a55ea2f0ddbeae07855741ace0ddca3a2b11d998478e8bb16c873e3d44"}
Oct 31 14:02:12.208170 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:12.208145    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:12.372520 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:12.372480    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:12.681152 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:12.681104    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:12.795543 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:12.795512    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a89282-5031-43e8-8a6c-f6d72b79cff1" path="/var/lib/kubelet/pods/e1a89282-5031-43e8-8a6c-f6d72b79cff1/volumes"
Oct 31 14:02:13.387443 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.387416    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"
Oct 31 14:02:13.525862 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.525823    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8922cd48-13c8-4172-9470-736ac36a46ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8922cd48-13c8-4172-9470-736ac36a46ed-maas-api-xks-ca\") pod \"8922cd48-13c8-4172-9470-736ac36a46ed\" (UID: \"8922cd48-13c8-4172-9470-736ac36a46ed\") "
Oct 31 14:02:13.526120 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.526011    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8922cd48-13c8-4172-9470-736ac36a46ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8922cd48-13c8-4172-9470-736ac36a46ed-maas-api-openshift-ca\") pod \"8922cd48-13c8-4172-9470-736ac36a46ed\" (UID: \"8922cd48-13c8-4172-9470-736ac36a46ed\") "
Oct 31 14:02:13.526120 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.526044    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8922cd48-13c8-4172-9470-736ac36a46ed-kube-api-access-xcthq\" (UniqueName: \"kubernetes.io/projected/8922cd48-13c8-4172-9470-736ac36a46ed-kube-api-access-xcthq\") pod \"8922cd48-13c8-4172-9470-736ac36a46ed\" (UID: \"8922cd48-13c8-4172-9470-736ac36a46ed\") "
Oct 31 14:02:13.526533 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.526498    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8922cd48-13c8-4172-9470-736ac36a46ed-maas-api-openshift-ca" pod "8922cd48-13c8-4172-9470-736ac36a46ed" (UID: "8922cd48-13c8-4172-9470-736ac36a46ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:13.529058 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.528516    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8922cd48-13c8-4172-9470-736ac36a46ed-kube-api-access-xcthq" pod "8922cd48-13c8-4172-9470-736ac36a46ed" (UID: "8922cd48-13c8-4172-9470-736ac36a46ed"). InnerVolumeSpecName "kube-api-access-xcthq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:13.529058 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.528564    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8922cd48-13c8-4172-9470-736ac36a46ed-maas-api-xks-ca" pod "8922cd48-13c8-4172-9470-736ac36a46ed" (UID: "8922cd48-13c8-4172-9470-736ac36a46ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:13.590816 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.590781    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:13.628159 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.628114    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8922cd48-13c8-4172-9470-736ac36a46ed-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:13.628159 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.628150    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8922cd48-13c8-4172-9470-736ac36a46ed-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:13.628159 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.628166    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcthq\" (UniqueName: \"kubernetes.io/projected/8922cd48-13c8-4172-9470-736ac36a46ed-kube-api-access-xcthq\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:13.799692 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.799663    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 14:02:13.812083 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.812035    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:13.945617 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.945583    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-workload-certs\") pod \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") "
Oct 31 14:02:13.945617 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.945623    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-workload-socket\") pod \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") "
Oct 31 14:02:13.945855 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.945698    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-token\" (UniqueName: \"kubernetes.io/projected/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-token\") pod \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") "
Oct 31 14:02:13.945855 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.945741    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-credential-socket\") pod \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") "
Oct 31 14:02:13.945855 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.945778    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-envoy\") pod \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") "
Oct 31 14:02:13.945855 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.945842    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-podinfo\") pod \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") "
Oct 31 14:02:13.946099 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.945943    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-workload-certs" pod "c1dfb224-4987-4a65-9cd3-56a3383aac7f" (UID: "c1dfb224-4987-4a65-9cd3-56a3383aac7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:13.946099 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.945979    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istiod-ca-cert\") pod \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") "
Oct 31 14:02:13.946099 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.946070    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-credential-socket" pod "c1dfb224-4987-4a65-9cd3-56a3383aac7f" (UID: "c1dfb224-4987-4a65-9cd3-56a3383aac7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:13.946099 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.946085    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-data\") pod \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") "
Oct 31 14:02:13.946323 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.946140    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1dfb224-4987-4a65-9cd3-56a3383aac7f-kube-api-access-mm6wt\" (UniqueName: \"kubernetes.io/projected/c1dfb224-4987-4a65-9cd3-56a3383aac7f-kube-api-access-mm6wt\") pod \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\" (UID: \"c1dfb224-4987-4a65-9cd3-56a3383aac7f\") "
Oct 31 14:02:13.946323 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.946087    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-workload-socket" pod "c1dfb224-4987-4a65-9cd3-56a3383aac7f" (UID: "c1dfb224-4987-4a65-9cd3-56a3383aac7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:13.946323 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.946301    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-data" pod "c1dfb224-4987-4a65-9cd3-56a3383aac7f" (UID: "c1dfb224-4987-4a65-9cd3-56a3383aac7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:13.946483 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.946379    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istiod-ca-cert" pod "c1dfb224-4987-4a65-9cd3-56a3383aac7f" (UID: "c1dfb224-4987-4a65-9cd3-56a3383aac7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:13.946754 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.946729    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:13.946858 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.946785    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:13.946938 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.946890    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:13.946992 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.946978    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:13.947044 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.946996    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:13.948403 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.948381    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-podinfo" pod "c1dfb224-4987-4a65-9cd3-56a3383aac7f" (UID: "c1dfb224-4987-4a65-9cd3-56a3383aac7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:02:13.948403 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.948399    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dfb224-4987-4a65-9cd3-56a3383aac7f-kube-api-access-mm6wt" pod "c1dfb224-4987-4a65-9cd3-56a3383aac7f" (UID: "c1dfb224-4987-4a65-9cd3-56a3383aac7f"). InnerVolumeSpecName "kube-api-access-mm6wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:13.948679 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.948652    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-envoy" pod "c1dfb224-4987-4a65-9cd3-56a3383aac7f" (UID: "c1dfb224-4987-4a65-9cd3-56a3383aac7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:13.948774 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:13.948676    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-token" pod "c1dfb224-4987-4a65-9cd3-56a3383aac7f" (UID: "c1dfb224-4987-4a65-9cd3-56a3383aac7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:14.038954 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.038854    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:14.039342 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.039322    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"]
Oct 31 14:02:14.047978 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.047945    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm6wt\" (UniqueName: \"kubernetes.io/projected/c1dfb224-4987-4a65-9cd3-56a3383aac7f-kube-api-access-mm6wt\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:14.047978 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.047980    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:14.048214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.047997    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:14.048214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.048009    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dfb224-4987-4a65-9cd3-56a3383aac7f-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:14.192731 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.192694    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"]
Oct 31 14:02:14.193029 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.193001    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" containerID="cri-o://35006d3b219885bd7a049ddcd535445877b705b76169fd8d56cbaf360f799119" gracePeriod=30
Oct 31 14:02:14.218952 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.218921    4428 generic.go:356] "Generic (PLEG): container finished" podID="c1dfb224-4987-4a65-9cd3-56a3383aac7f" containerID="8cc88ea0301d428bd54c91620f0c7e1e87b92176c21a9d8de5e9ef02862128e3" exitCode=0
Oct 31 14:02:14.219131 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.218962    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2" event={"ID":"c1dfb224-4987-4a65-9cd3-56a3383aac7f","Type":"ContainerDied","Data":"8cc88ea0301d428bd54c91620f0c7e1e87b92176c21a9d8de5e9ef02862128e3"}
Oct 31 14:02:14.219131 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.218986    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"
Oct 31 14:02:14.219131 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.219037    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2" event={"ID":"c1dfb224-4987-4a65-9cd3-56a3383aac7f","Type":"ContainerDied","Data":"a72662f2f2f69e261656c4a9fe3ea8dbe1ba61216ac404411fdb2ec5b65f393b"}
Oct 31 14:02:14.219131 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.219071    4428 scope.go:122] "RemoveContainer" containerID="8cc88ea0301d428bd54c91620f0c7e1e87b92176c21a9d8de5e9ef02862128e3"
Oct 31 14:02:14.220710 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.220684    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"
Oct 31 14:02:14.220841 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.220742    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf" event={"ID":"8922cd48-13c8-4172-9470-736ac36a46ed","Type":"ContainerDied","Data":"f2a0994da8b215d95286b258fc4825854b07c6458b2c32a2be9dbadee6432f88"}
Oct 31 14:02:14.220841 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.220793    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2a0994da8b215d95286b258fc4825854b07c6458b2c32a2be9dbadee6432f88"
Oct 31 14:02:14.231749 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.231732    4428 scope.go:122] "RemoveContainer" containerID="8cc88ea0301d428bd54c91620f0c7e1e87b92176c21a9d8de5e9ef02862128e3"
Oct 31 14:02:14.232175 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:14.232150    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc88ea0301d428bd54c91620f0c7e1e87b92176c21a9d8de5e9ef02862128e3\": container with ID starting with 8cc88ea0301d428bd54c91620f0c7e1e87b92176c21a9d8de5e9ef02862128e3 not found: ID does not exist" containerID="8cc88ea0301d428bd54c91620f0c7e1e87b92176c21a9d8de5e9ef02862128e3"
Oct 31 14:02:14.232300 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.232181    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc88ea0301d428bd54c91620f0c7e1e87b92176c21a9d8de5e9ef02862128e3"} err="failed to get container status \"8cc88ea0301d428bd54c91620f0c7e1e87b92176c21a9d8de5e9ef02862128e3\": rpc error: code = NotFound desc = could not find container \"8cc88ea0301d428bd54c91620f0c7e1e87b92176c21a9d8de5e9ef02862128e3\": container with ID starting with 8cc88ea0301d428bd54c91620f0c7e1e87b92176c21a9d8de5e9ef02862128e3 not found: ID does not exist"
Oct 31 14:02:14.263132 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.262760    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:14.263913 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.263889    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"]
Oct 31 14:02:14.267358 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.267330    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-hvcs2"]
Oct 31 14:02:14.612773 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.612688    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:14.756673 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:14.756639    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1dfb224-4987-4a65-9cd3-56a3383aac7f" path="/var/lib/kubelet/pods/c1dfb224-4987-4a65-9cd3-56a3383aac7f/volumes"
Oct 31 14:02:15.025700 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:15.025663    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:15.206354 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:15.206315    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:15.775969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:15.775933    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:16.041993 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:16.041844    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:16.438971 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:16.438859    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:16.932112 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:16.932030    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:17.101153 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:17.101118    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:17.882772 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:17.882735    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:18.019004 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.018977    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"
Oct 31 14:02:18.122223 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.122087    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"
Oct 31 14:02:18.124956 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.124932    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"
Oct 31 14:02:18.125090 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.125017    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"
Oct 31 14:02:18.138345 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.138313    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/461e6055-9080-4aab-8f66-43139b5b9cc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461e6055-9080-4aab-8f66-43139b5b9cc4-plugins-config-volume\") pod \"461e6055-9080-4aab-8f66-43139b5b9cc4\" (UID: \"461e6055-9080-4aab-8f66-43139b5b9cc4\") "
Oct 31 14:02:18.138532 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.138518    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/461e6055-9080-4aab-8f66-43139b5b9cc4-kube-api-access-c9zks\" (UniqueName: \"kubernetes.io/projected/461e6055-9080-4aab-8f66-43139b5b9cc4-kube-api-access-c9zks\") pod \"461e6055-9080-4aab-8f66-43139b5b9cc4\" (UID: \"461e6055-9080-4aab-8f66-43139b5b9cc4\") "
Oct 31 14:02:18.138735 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.138712    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/461e6055-9080-4aab-8f66-43139b5b9cc4-plugins-config-volume" pod "461e6055-9080-4aab-8f66-43139b5b9cc4" (UID: "461e6055-9080-4aab-8f66-43139b5b9cc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:18.139071 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.139047    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461e6055-9080-4aab-8f66-43139b5b9cc4-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:18.140848 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.140822    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461e6055-9080-4aab-8f66-43139b5b9cc4-kube-api-access-c9zks" pod "461e6055-9080-4aab-8f66-43139b5b9cc4" (UID: "461e6055-9080-4aab-8f66-43139b5b9cc4"). InnerVolumeSpecName "kube-api-access-c9zks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:18.240366 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.240278    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bab754b1-4f25-4274-a326-579b5326b39a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bab754b1-4f25-4274-a326-579b5326b39a-plugins-config-volume\") pod \"bab754b1-4f25-4274-a326-579b5326b39a\" (UID: \"bab754b1-4f25-4274-a326-579b5326b39a\") "
Oct 31 14:02:18.240660 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.240637    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab754b1-4f25-4274-a326-579b5326b39a-kube-api-access-vjklx\" (UniqueName: \"kubernetes.io/projected/bab754b1-4f25-4274-a326-579b5326b39a-kube-api-access-vjklx\") pod \"bab754b1-4f25-4274-a326-579b5326b39a\" (UID: \"bab754b1-4f25-4274-a326-579b5326b39a\") "
Oct 31 14:02:18.240752 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.240666    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab754b1-4f25-4274-a326-579b5326b39a-plugins-config-volume" pod "bab754b1-4f25-4274-a326-579b5326b39a" (UID: "bab754b1-4f25-4274-a326-579b5326b39a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:18.241671 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.241609    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bab754b1-4f25-4274-a326-579b5326b39a-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:18.241671 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.241639    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9zks\" (UniqueName: \"kubernetes.io/projected/461e6055-9080-4aab-8f66-43139b5b9cc4-kube-api-access-c9zks\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:18.243975 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.243947    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab754b1-4f25-4274-a326-579b5326b39a-kube-api-access-vjklx" pod "bab754b1-4f25-4274-a326-579b5326b39a" (UID: "bab754b1-4f25-4274-a326-579b5326b39a"). InnerVolumeSpecName "kube-api-access-vjklx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:18.266948 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.266847    4428 generic.go:356] "Generic (PLEG): container finished" podID="bab754b1-4f25-4274-a326-579b5326b39a" containerID="30d972a1ed8bc132c0ed4a9aa8ce5e45564c4be675592c69bc7fff839b243452" exitCode=0
Oct 31 14:02:18.267286 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.266956    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"
Oct 31 14:02:18.267286 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.267014    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f" event={"ID":"bab754b1-4f25-4274-a326-579b5326b39a","Type":"ContainerDied","Data":"30d972a1ed8bc132c0ed4a9aa8ce5e45564c4be675592c69bc7fff839b243452"}
Oct 31 14:02:18.267286 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.267067    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f" event={"ID":"bab754b1-4f25-4274-a326-579b5326b39a","Type":"ContainerDied","Data":"dcfd111256fdd5fc98b752da5ac969914fe924e7abb1f6675a649a67bd52bc59"}
Oct 31 14:02:18.267286 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.267098    4428 scope.go:122] "RemoveContainer" containerID="30d972a1ed8bc132c0ed4a9aa8ce5e45564c4be675592c69bc7fff839b243452"
Oct 31 14:02:18.270013 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.269851    4428 generic.go:356] "Generic (PLEG): container finished" podID="461e6055-9080-4aab-8f66-43139b5b9cc4" containerID="7b681ae63c8fd118ac450541b1dfce433edb2f63b8f9a076af9dcf0bab675d5a" exitCode=0
Oct 31 14:02:18.270013 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.270000    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8" event={"ID":"461e6055-9080-4aab-8f66-43139b5b9cc4","Type":"ContainerDied","Data":"7b681ae63c8fd118ac450541b1dfce433edb2f63b8f9a076af9dcf0bab675d5a"}
Oct 31 14:02:18.270202 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.270051    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8" event={"ID":"461e6055-9080-4aab-8f66-43139b5b9cc4","Type":"ContainerDied","Data":"5ee127456fca8d07a22c9856ea8943f29305754c59bc6ca92f95b959c9de4a44"}
Oct 31 14:02:18.270202 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.269981    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"
Oct 31 14:02:18.284188 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.284153    4428 scope.go:122] "RemoveContainer" containerID="30d972a1ed8bc132c0ed4a9aa8ce5e45564c4be675592c69bc7fff839b243452"
Oct 31 14:02:18.285099 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:18.284576    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d972a1ed8bc132c0ed4a9aa8ce5e45564c4be675592c69bc7fff839b243452\": container with ID starting with 30d972a1ed8bc132c0ed4a9aa8ce5e45564c4be675592c69bc7fff839b243452 not found: ID does not exist" containerID="30d972a1ed8bc132c0ed4a9aa8ce5e45564c4be675592c69bc7fff839b243452"
Oct 31 14:02:18.285099 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.284625    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d972a1ed8bc132c0ed4a9aa8ce5e45564c4be675592c69bc7fff839b243452"} err="failed to get container status \"30d972a1ed8bc132c0ed4a9aa8ce5e45564c4be675592c69bc7fff839b243452\": rpc error: code = NotFound desc = could not find container \"30d972a1ed8bc132c0ed4a9aa8ce5e45564c4be675592c69bc7fff839b243452\": container with ID starting with 30d972a1ed8bc132c0ed4a9aa8ce5e45564c4be675592c69bc7fff839b243452 not found: ID does not exist"
Oct 31 14:02:18.285099 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.284647    4428 generic.go:356] "Generic (PLEG): container finished" podID="2cba5737-a6a1-4c1c-8b1d-24f1f7162a60" containerID="41b23ae38f5092c43300be2a3b088fb8d20417efce971f7d1772bd09bbcb3219" exitCode=0
Oct 31 14:02:18.285099 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.284658    4428 scope.go:122] "RemoveContainer" containerID="7b681ae63c8fd118ac450541b1dfce433edb2f63b8f9a076af9dcf0bab675d5a"
Oct 31 14:02:18.285099 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.284683    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v" event={"ID":"2cba5737-a6a1-4c1c-8b1d-24f1f7162a60","Type":"ContainerDied","Data":"41b23ae38f5092c43300be2a3b088fb8d20417efce971f7d1772bd09bbcb3219"}
Oct 31 14:02:18.318401 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.318362    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"]
Oct 31 14:02:18.328161 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.328122    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-14ba5a-845bd58c69-zkbf8"]
Oct 31 14:02:18.330307 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.330272    4428 scope.go:122] "RemoveContainer" containerID="7b681ae63c8fd118ac450541b1dfce433edb2f63b8f9a076af9dcf0bab675d5a"
Oct 31 14:02:18.330887 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:18.330849    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b681ae63c8fd118ac450541b1dfce433edb2f63b8f9a076af9dcf0bab675d5a\": container with ID starting with 7b681ae63c8fd118ac450541b1dfce433edb2f63b8f9a076af9dcf0bab675d5a not found: ID does not exist" containerID="7b681ae63c8fd118ac450541b1dfce433edb2f63b8f9a076af9dcf0bab675d5a"
Oct 31 14:02:18.330983 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.330902    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b681ae63c8fd118ac450541b1dfce433edb2f63b8f9a076af9dcf0bab675d5a"} err="failed to get container status \"7b681ae63c8fd118ac450541b1dfce433edb2f63b8f9a076af9dcf0bab675d5a\": rpc error: code = NotFound desc = could not find container \"7b681ae63c8fd118ac450541b1dfce433edb2f63b8f9a076af9dcf0bab675d5a\": container with ID starting with 7b681ae63c8fd118ac450541b1dfce433edb2f63b8f9a076af9dcf0bab675d5a not found: ID does not exist"
Oct 31 14:02:18.335533 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.335511    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"
Oct 31 14:02:18.343168 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.343128    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjklx\" (UniqueName: \"kubernetes.io/projected/bab754b1-4f25-4274-a326-579b5326b39a-kube-api-access-vjklx\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:18.347275 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.347252    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"]
Oct 31 14:02:18.353807 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.353784    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-14ba5a-6dc47df64c-sf74f"]
Oct 31 14:02:18.444385 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.444353    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cba5737-a6a1-4c1c-8b1d-24f1f7162a60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cba5737-a6a1-4c1c-8b1d-24f1f7162a60-plugins-config-volume\") pod \"2cba5737-a6a1-4c1c-8b1d-24f1f7162a60\" (UID: \"2cba5737-a6a1-4c1c-8b1d-24f1f7162a60\") "
Oct 31 14:02:18.444602 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.444444    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cba5737-a6a1-4c1c-8b1d-24f1f7162a60-kube-api-access-r4ltl\" (UniqueName: \"kubernetes.io/projected/2cba5737-a6a1-4c1c-8b1d-24f1f7162a60-kube-api-access-r4ltl\") pod \"2cba5737-a6a1-4c1c-8b1d-24f1f7162a60\" (UID: \"2cba5737-a6a1-4c1c-8b1d-24f1f7162a60\") "
Oct 31 14:02:18.444735 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.444709    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cba5737-a6a1-4c1c-8b1d-24f1f7162a60-plugins-config-volume" pod "2cba5737-a6a1-4c1c-8b1d-24f1f7162a60" (UID: "2cba5737-a6a1-4c1c-8b1d-24f1f7162a60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:18.444976 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.444959    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cba5737-a6a1-4c1c-8b1d-24f1f7162a60-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:18.446769 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.446748    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cba5737-a6a1-4c1c-8b1d-24f1f7162a60-kube-api-access-r4ltl" pod "2cba5737-a6a1-4c1c-8b1d-24f1f7162a60" (UID: "2cba5737-a6a1-4c1c-8b1d-24f1f7162a60"). InnerVolumeSpecName "kube-api-access-r4ltl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:18.546016 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.545910    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4ltl\" (UniqueName: \"kubernetes.io/projected/2cba5737-a6a1-4c1c-8b1d-24f1f7162a60-kube-api-access-r4ltl\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:18.742677 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:18.742641    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:02:18.756047 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.756007    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="461e6055-9080-4aab-8f66-43139b5b9cc4" path="/var/lib/kubelet/pods/461e6055-9080-4aab-8f66-43139b5b9cc4/volumes"
Oct 31 14:02:18.756548 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.756525    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab754b1-4f25-4274-a326-579b5326b39a" path="/var/lib/kubelet/pods/bab754b1-4f25-4274-a326-579b5326b39a/volumes"
Oct 31 14:02:18.875955 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:18.872172    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:19.259543 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.259503    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"]
Oct 31 14:02:19.310817 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.310780    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v" event={"ID":"2cba5737-a6a1-4c1c-8b1d-24f1f7162a60","Type":"ContainerDied","Data":"5fec98c22105d49a94629aa2522496cb09fb0988bad34016b4fed208e34ee011"}
Oct 31 14:02:19.311514 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.311469    4428 scope.go:122] "RemoveContainer" containerID="41b23ae38f5092c43300be2a3b088fb8d20417efce971f7d1772bd09bbcb3219"
Oct 31 14:02:19.311665 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.311647    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"
Oct 31 14:02:19.336421 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.336313    4428 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 14:02:19.336421 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.336383    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 14:02:19.336911 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.336883    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"]
Oct 31 14:02:19.341469 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.341442    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-15dc2f-fdc4cf6c-zgv5v"]
Oct 31 14:02:19.363518 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.363484    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5" podUID="e8641866-a4d3-4c0d-8d8a-dd25e8572139" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:19.416567 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.416533    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:19.485972 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.485939    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 14:02:19.580936 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.573582    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-podinfo\") pod \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") "
Oct 31 14:02:19.580936 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.573669    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-data\") pod \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") "
Oct 31 14:02:19.580936 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.573715    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-workload-socket\") pod \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") "
Oct 31 14:02:19.580936 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.573746    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-workload-certs\") pod \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") "
Oct 31 14:02:19.580936 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.573791    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-credential-socket\") pod \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") "
Oct 31 14:02:19.580936 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.573920    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istiod-ca-cert\") pod \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") "
Oct 31 14:02:19.580936 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.574005    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-kube-api-access-l9bmx\" (UniqueName: \"kubernetes.io/projected/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-kube-api-access-l9bmx\") pod \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") "
Oct 31 14:02:19.580936 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.574137    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-envoy\") pod \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") "
Oct 31 14:02:19.580936 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.574212    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-token\" (UniqueName: \"kubernetes.io/projected/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-token\") pod \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\" (UID: \"ffbf53aa-dea2-46cb-8bb6-9164ebffb925\") "
Oct 31 14:02:19.598909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.586496    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-workload-socket" pod "ffbf53aa-dea2-46cb-8bb6-9164ebffb925" (UID: "ffbf53aa-dea2-46cb-8bb6-9164ebffb925"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:19.598909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.587342    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istiod-ca-cert" pod "ffbf53aa-dea2-46cb-8bb6-9164ebffb925" (UID: "ffbf53aa-dea2-46cb-8bb6-9164ebffb925"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:19.598909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.587646    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-workload-certs" pod "ffbf53aa-dea2-46cb-8bb6-9164ebffb925" (UID: "ffbf53aa-dea2-46cb-8bb6-9164ebffb925"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:19.598909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.587926    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-credential-socket" pod "ffbf53aa-dea2-46cb-8bb6-9164ebffb925" (UID: "ffbf53aa-dea2-46cb-8bb6-9164ebffb925"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:19.609728 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.602441    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-kube-api-access-l9bmx" pod "ffbf53aa-dea2-46cb-8bb6-9164ebffb925" (UID: "ffbf53aa-dea2-46cb-8bb6-9164ebffb925"). InnerVolumeSpecName "kube-api-access-l9bmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:19.609728 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.605304    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-envoy" pod "ffbf53aa-dea2-46cb-8bb6-9164ebffb925" (UID: "ffbf53aa-dea2-46cb-8bb6-9164ebffb925"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:19.609728 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.605548    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-data" pod "ffbf53aa-dea2-46cb-8bb6-9164ebffb925" (UID: "ffbf53aa-dea2-46cb-8bb6-9164ebffb925"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:19.609728 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.605808    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:19.648752 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.648323    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-token" pod "ffbf53aa-dea2-46cb-8bb6-9164ebffb925" (UID: "ffbf53aa-dea2-46cb-8bb6-9164ebffb925"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:19.652313 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.650160    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-podinfo" pod "ffbf53aa-dea2-46cb-8bb6-9164ebffb925" (UID: "ffbf53aa-dea2-46cb-8bb6-9164ebffb925"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:02:19.675614 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.675583    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:19.675614 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.675612    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:19.675859 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.675626    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:19.675859 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.675640    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:19.675859 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.675654    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:19.675859 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.675665    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:19.675859 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.675676    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:19.675859 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.675688    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:19.675859 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.675702    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9bmx\" (UniqueName: \"kubernetes.io/projected/ffbf53aa-dea2-46cb-8bb6-9164ebffb925-kube-api-access-l9bmx\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:19.847061 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.846972    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"]
Oct 31 14:02:19.870467 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.870429    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"]
Oct 31 14:02:19.987516 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:19.987486    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:20.241795 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:20.241746    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:20.321324 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:20.321287    4428 generic.go:356] "Generic (PLEG): container finished" podID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerID="35006d3b219885bd7a049ddcd535445877b705b76169fd8d56cbaf360f799119" exitCode=0
Oct 31 14:02:20.321809 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:20.321340    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" event={"ID":"ffbf53aa-dea2-46cb-8bb6-9164ebffb925","Type":"ContainerDied","Data":"35006d3b219885bd7a049ddcd535445877b705b76169fd8d56cbaf360f799119"}
Oct 31 14:02:20.321809 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:20.321346    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"
Oct 31 14:02:20.321809 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:20.321383    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd" event={"ID":"ffbf53aa-dea2-46cb-8bb6-9164ebffb925","Type":"ContainerDied","Data":"580ae4372e90dd75bd383351d8af39485ec0744c8e17e68de27f55f661055f55"}
Oct 31 14:02:20.321809 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:20.321403    4428 scope.go:122] "RemoveContainer" containerID="35006d3b219885bd7a049ddcd535445877b705b76169fd8d56cbaf360f799119"
Oct 31 14:02:20.333474 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:20.333448    4428 scope.go:122] "RemoveContainer" containerID="35006d3b219885bd7a049ddcd535445877b705b76169fd8d56cbaf360f799119"
Oct 31 14:02:20.333770 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:20.333749    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35006d3b219885bd7a049ddcd535445877b705b76169fd8d56cbaf360f799119\": container with ID starting with 35006d3b219885bd7a049ddcd535445877b705b76169fd8d56cbaf360f799119 not found: ID does not exist" containerID="35006d3b219885bd7a049ddcd535445877b705b76169fd8d56cbaf360f799119"
Oct 31 14:02:20.333825 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:20.333780    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35006d3b219885bd7a049ddcd535445877b705b76169fd8d56cbaf360f799119"} err="failed to get container status \"35006d3b219885bd7a049ddcd535445877b705b76169fd8d56cbaf360f799119\": rpc error: code = NotFound desc = could not find container \"35006d3b219885bd7a049ddcd535445877b705b76169fd8d56cbaf360f799119\": container with ID starting with 35006d3b219885bd7a049ddcd535445877b705b76169fd8d56cbaf360f799119 not found: ID does not exist"
Oct 31 14:02:20.348045 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:20.348016    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"]
Oct 31 14:02:20.359487 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:20.359448    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-4q6cd"]
Oct 31 14:02:20.487144 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:20.487074    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:20.646137 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:20.646041    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:20.745922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:20.744293    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"]
Oct 31 14:02:20.745922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:20.744626    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n" podUID="b9fb27a2-762a-4256-8c53-89a6bb50e654" containerName="istio-proxy" containerID="cri-o://9ffddf3cb81a87a3a21b322f62192d392287e6938ab5825ed15d2eae06344e3a" gracePeriod=30
Oct 31 14:02:20.759347 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:20.759315    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cba5737-a6a1-4c1c-8b1d-24f1f7162a60" path="/var/lib/kubelet/pods/2cba5737-a6a1-4c1c-8b1d-24f1f7162a60/volumes"
Oct 31 14:02:20.759816 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:20.759803    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" path="/var/lib/kubelet/pods/ffbf53aa-dea2-46cb-8bb6-9164ebffb925/volumes"
Oct 31 14:02:20.857898 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:20.845135    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:21.012027 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.011912    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:21.280970 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.280830    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:21.366589 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.366554    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"]
Oct 31 14:02:21.367189 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367174    4428 cpu_manager.go:414] "containerMap: removing container" podUID="c1dfb224-4987-4a65-9cd3-56a3383aac7f" containerName="istio-proxy"
Oct 31 14:02:21.367249 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367190    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1dfb224-4987-4a65-9cd3-56a3383aac7f" containerName="istio-proxy"
Oct 31 14:02:21.367249 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367204    4428 cpu_manager.go:414] "containerMap: removing container" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy"
Oct 31 14:02:21.367249 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367209    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy"
Oct 31 14:02:21.367249 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367215    4428 cpu_manager.go:414] "containerMap: removing container" podUID="2cba5737-a6a1-4c1c-8b1d-24f1f7162a60" containerName="payload-pre-processing"
Oct 31 14:02:21.367249 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367221    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cba5737-a6a1-4c1c-8b1d-24f1f7162a60" containerName="payload-pre-processing"
Oct 31 14:02:21.367249 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367249    4428 cpu_manager.go:414] "containerMap: removing container" podUID="461e6055-9080-4aab-8f66-43139b5b9cc4" containerName="payload-processing"
Oct 31 14:02:21.367451 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367253    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461e6055-9080-4aab-8f66-43139b5b9cc4" containerName="payload-processing"
Oct 31 14:02:21.367451 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367268    4428 cpu_manager.go:414] "containerMap: removing container" podUID="8922cd48-13c8-4172-9470-736ac36a46ed" containerName="revoke-keys"
Oct 31 14:02:21.367451 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367273    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8922cd48-13c8-4172-9470-736ac36a46ed" containerName="revoke-keys"
Oct 31 14:02:21.367451 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367284    4428 cpu_manager.go:414] "containerMap: removing container" podUID="bab754b1-4f25-4274-a326-579b5326b39a" containerName="payload-pre-processing"
Oct 31 14:02:21.367451 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367289    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab754b1-4f25-4274-a326-579b5326b39a" containerName="payload-pre-processing"
Oct 31 14:02:21.367451 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367365    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8922cd48-13c8-4172-9470-736ac36a46ed" containerName="revoke-keys"
Oct 31 14:02:21.367451 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367375    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab754b1-4f25-4274-a326-579b5326b39a" containerName="payload-pre-processing"
Oct 31 14:02:21.367451 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367384    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1dfb224-4987-4a65-9cd3-56a3383aac7f" containerName="istio-proxy"
Oct 31 14:02:21.367451 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367389    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffbf53aa-dea2-46cb-8bb6-9164ebffb925" containerName="istio-proxy"
Oct 31 14:02:21.367451 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367394    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cba5737-a6a1-4c1c-8b1d-24f1f7162a60" containerName="payload-pre-processing"
Oct 31 14:02:21.367451 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.367404    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="461e6055-9080-4aab-8f66-43139b5b9cc4" containerName="payload-processing"
Oct 31 14:02:21.372606 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.372585    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"
Oct 31 14:02:21.381810 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.381786    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"]
Oct 31 14:02:21.402473 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.402445    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:02:21.402758 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.402740    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:02:21.407530 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.407506    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg\" (UID: \"d664724b-e6a5-4cad-8317-b97fcfa2e0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"
Oct 31 14:02:21.407668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.407589    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg\" (UID: \"d664724b-e6a5-4cad-8317-b97fcfa2e0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"
Oct 31 14:02:21.407810 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.407785    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7w52\" (UniqueName: \"kubernetes.io/projected/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-kube-api-access-c7w52\") pod \"maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg\" (UID: \"d664724b-e6a5-4cad-8317-b97fcfa2e0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"
Oct 31 14:02:21.509605 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.509554    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7w52\" (UniqueName: \"kubernetes.io/projected/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-kube-api-access-c7w52\") pod \"maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg\" (UID: \"d664724b-e6a5-4cad-8317-b97fcfa2e0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"
Oct 31 14:02:21.509798 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.509649    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg\" (UID: \"d664724b-e6a5-4cad-8317-b97fcfa2e0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"
Oct 31 14:02:21.509798 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.509708    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg\" (UID: \"d664724b-e6a5-4cad-8317-b97fcfa2e0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"
Oct 31 14:02:21.510351 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.510317    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg\" (UID: \"d664724b-e6a5-4cad-8317-b97fcfa2e0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"
Oct 31 14:02:21.512332 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.512307    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg\" (UID: \"d664724b-e6a5-4cad-8317-b97fcfa2e0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"
Oct 31 14:02:21.530740 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.530712    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7w52\" (UniqueName: \"kubernetes.io/projected/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-kube-api-access-c7w52\") pod \"maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg\" (UID: \"d664724b-e6a5-4cad-8317-b97fcfa2e0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"
Oct 31 14:02:21.694742 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.694652    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"
Oct 31 14:02:21.769894 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.769719    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"]
Oct 31 14:02:21.777908 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.777005    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"
Oct 31 14:02:21.788413 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.788076    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"]
Oct 31 14:02:21.814458 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.814424    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zv4r\" (UniqueName: \"kubernetes.io/projected/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-kube-api-access-4zv4r\") pod \"maas-api-revoke-sub-baaaafb4-nfkrp\" (UID: \"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"
Oct 31 14:02:21.814642 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.814597    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baaaafb4-nfkrp\" (UID: \"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"
Oct 31 14:02:21.814738 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.814689    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baaaafb4-nfkrp\" (UID: \"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"
Oct 31 14:02:21.916054 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.916018    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baaaafb4-nfkrp\" (UID: \"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"
Oct 31 14:02:21.916233 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.916131    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zv4r\" (UniqueName: \"kubernetes.io/projected/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-kube-api-access-4zv4r\") pod \"maas-api-revoke-sub-baaaafb4-nfkrp\" (UID: \"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"
Oct 31 14:02:21.916233 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.916226    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baaaafb4-nfkrp\" (UID: \"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"
Oct 31 14:02:21.916818 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.916795    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baaaafb4-nfkrp\" (UID: \"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"
Oct 31 14:02:21.918685 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.918657    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baaaafb4-nfkrp\" (UID: \"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"
Oct 31 14:02:21.929680 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.929659    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zv4r\" (UniqueName: \"kubernetes.io/projected/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-kube-api-access-4zv4r\") pod \"maas-api-revoke-sub-baaaafb4-nfkrp\" (UID: \"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"
Oct 31 14:02:21.981953 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:21.981920    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"]
Oct 31 14:02:21.983081 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:02:21.983053    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd664724b_e6a5_4cad_8317_b97fcfa2e0fe.slice/crio-39003101f24393c63a08436b33238d243e88fa559467c0e6cc105e274c86f275 WatchSource:0}: Error finding container 39003101f24393c63a08436b33238d243e88fa559467c0e6cc105e274c86f275: Status 404 returned error can't find the container with id 39003101f24393c63a08436b33238d243e88fa559467c0e6cc105e274c86f275
Oct 31 14:02:22.103395 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:22.103365    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"
Oct 31 14:02:22.307306 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:22.307266    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:22.340156 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:22.340112    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg" event={"ID":"d664724b-e6a5-4cad-8317-b97fcfa2e0fe","Type":"ContainerStarted","Data":"5584440e5b9fe1f5d1d91cb1b99b855a00e4eaf4fe4d35d893d85f4d7abc4670"}
Oct 31 14:02:22.340354 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:22.340180    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg" event={"ID":"d664724b-e6a5-4cad-8317-b97fcfa2e0fe","Type":"ContainerStarted","Data":"39003101f24393c63a08436b33238d243e88fa559467c0e6cc105e274c86f275"}
Oct 31 14:02:22.365489 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:22.365437    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg" podStartSLOduration=1.365421259 podStartE2EDuration="1.365421259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:22.363492625 +0000 UTC m=+3539.761841251" watchObservedRunningTime="2026-10-31 14:02:22.365421259 +0000 UTC m=+3539.763769916"
Oct 31 14:02:22.400628 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:22.400593    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"]
Oct 31 14:02:22.402023 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:02:22.401993    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfd0f1c_8e15_4d6c_bdef_34581cbe28d6.slice/crio-b5baf0672a4fb0efd292b1956d80bf974c2bbbd5a1f4751b441893f6cb0b6231 WatchSource:0}: Error finding container b5baf0672a4fb0efd292b1956d80bf974c2bbbd5a1f4751b441893f6cb0b6231: Status 404 returned error can't find the container with id b5baf0672a4fb0efd292b1956d80bf974c2bbbd5a1f4751b441893f6cb0b6231
Oct 31 14:02:22.897069 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:22.896244    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"]
Oct 31 14:02:22.897069 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:22.896583    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws" podUID="2eacbe93-226a-4def-96fe-b8409cc76f3b" containerName="maas-api" containerID="cri-o://6d07a417a529943aeef6e945444deda779c3c6c95a53e3b20af6386508946cd3" gracePeriod=30
Oct 31 14:02:23.008430 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.008393    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:23.243292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.243265    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 14:02:23.249283 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.249256    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:23.278935 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.276503    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eacbe93-226a-4def-96fe-b8409cc76f3b-kube-api-access-c5m6p\" (UniqueName: \"kubernetes.io/projected/2eacbe93-226a-4def-96fe-b8409cc76f3b-kube-api-access-c5m6p\") pod \"2eacbe93-226a-4def-96fe-b8409cc76f3b\" (UID: \"2eacbe93-226a-4def-96fe-b8409cc76f3b\") "
Oct 31 14:02:23.278935 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.276618    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eacbe93-226a-4def-96fe-b8409cc76f3b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eacbe93-226a-4def-96fe-b8409cc76f3b-maas-api-tls\") pod \"2eacbe93-226a-4def-96fe-b8409cc76f3b\" (UID: \"2eacbe93-226a-4def-96fe-b8409cc76f3b\") "
Oct 31 14:02:23.278935 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.276822    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eacbe93-226a-4def-96fe-b8409cc76f3b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eacbe93-226a-4def-96fe-b8409cc76f3b-openshift-service-ca\") pod \"2eacbe93-226a-4def-96fe-b8409cc76f3b\" (UID: \"2eacbe93-226a-4def-96fe-b8409cc76f3b\") "
Oct 31 14:02:23.278935 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.277237    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eacbe93-226a-4def-96fe-b8409cc76f3b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eacbe93-226a-4def-96fe-b8409cc76f3b-metrics-certs\") pod \"2eacbe93-226a-4def-96fe-b8409cc76f3b\" (UID: \"2eacbe93-226a-4def-96fe-b8409cc76f3b\") "
Oct 31 14:02:23.278935 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.277799    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"]
Oct 31 14:02:23.278935 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.278208    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eacbe93-226a-4def-96fe-b8409cc76f3b-openshift-service-ca" pod "2eacbe93-226a-4def-96fe-b8409cc76f3b" (UID: "2eacbe93-226a-4def-96fe-b8409cc76f3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:23.280752 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.280721    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-5e8f29-gx4vf"]
Oct 31 14:02:23.281179 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.281153    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eacbe93-226a-4def-96fe-b8409cc76f3b-kube-api-access-c5m6p" pod "2eacbe93-226a-4def-96fe-b8409cc76f3b" (UID: "2eacbe93-226a-4def-96fe-b8409cc76f3b"). InnerVolumeSpecName "kube-api-access-c5m6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:23.281345 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.281315    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eacbe93-226a-4def-96fe-b8409cc76f3b-metrics-certs" pod "2eacbe93-226a-4def-96fe-b8409cc76f3b" (UID: "2eacbe93-226a-4def-96fe-b8409cc76f3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:23.282850 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.282791    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eacbe93-226a-4def-96fe-b8409cc76f3b-maas-api-tls" pod "2eacbe93-226a-4def-96fe-b8409cc76f3b" (UID: "2eacbe93-226a-4def-96fe-b8409cc76f3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:23.348175 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.348144    4428 generic.go:356] "Generic (PLEG): container finished" podID="4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6" containerID="3011b44c1a90a3f60c8b17d31b4ad11c6a379c715b5e02d8e4f7ea322f6d0fa8" exitCode=0
Oct 31 14:02:23.348335 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.348242    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp" event={"ID":"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6","Type":"ContainerDied","Data":"3011b44c1a90a3f60c8b17d31b4ad11c6a379c715b5e02d8e4f7ea322f6d0fa8"}
Oct 31 14:02:23.348335 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.348291    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp" event={"ID":"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6","Type":"ContainerStarted","Data":"b5baf0672a4fb0efd292b1956d80bf974c2bbbd5a1f4751b441893f6cb0b6231"}
Oct 31 14:02:23.349904 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.349836    4428 generic.go:356] "Generic (PLEG): container finished" podID="d664724b-e6a5-4cad-8317-b97fcfa2e0fe" containerID="5584440e5b9fe1f5d1d91cb1b99b855a00e4eaf4fe4d35d893d85f4d7abc4670" exitCode=0
Oct 31 14:02:23.350038 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.349914    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg" event={"ID":"d664724b-e6a5-4cad-8317-b97fcfa2e0fe","Type":"ContainerDied","Data":"5584440e5b9fe1f5d1d91cb1b99b855a00e4eaf4fe4d35d893d85f4d7abc4670"}
Oct 31 14:02:23.351491 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.351466    4428 generic.go:356] "Generic (PLEG): container finished" podID="2eacbe93-226a-4def-96fe-b8409cc76f3b" containerID="6d07a417a529943aeef6e945444deda779c3c6c95a53e3b20af6386508946cd3" exitCode=0
Oct 31 14:02:23.351627 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.351510    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"
Oct 31 14:02:23.351627 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.351525    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws" event={"ID":"2eacbe93-226a-4def-96fe-b8409cc76f3b","Type":"ContainerDied","Data":"6d07a417a529943aeef6e945444deda779c3c6c95a53e3b20af6386508946cd3"}
Oct 31 14:02:23.351627 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.351564    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws" event={"ID":"2eacbe93-226a-4def-96fe-b8409cc76f3b","Type":"ContainerDied","Data":"2b6735b8b8aee68f86bc3d8c6064305d1e5d4db1bb72336d3ad98f2d6d1bc0e2"}
Oct 31 14:02:23.351627 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.351589    4428 scope.go:122] "RemoveContainer" containerID="6d07a417a529943aeef6e945444deda779c3c6c95a53e3b20af6386508946cd3"
Oct 31 14:02:23.378949 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.378909    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eacbe93-226a-4def-96fe-b8409cc76f3b-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:23.378949 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.378943    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eacbe93-226a-4def-96fe-b8409cc76f3b-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:23.378949 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.378953    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5m6p\" (UniqueName: \"kubernetes.io/projected/2eacbe93-226a-4def-96fe-b8409cc76f3b-kube-api-access-c5m6p\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:23.379207 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.378963    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eacbe93-226a-4def-96fe-b8409cc76f3b-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:23.435143 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.435091    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"]
Oct 31 14:02:23.439841 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.439774    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:23.440248 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.440222    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5e8f29-84f4cd7646-r7cws"]
Oct 31 14:02:23.443668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.443644    4428 scope.go:122] "RemoveContainer" containerID="6d07a417a529943aeef6e945444deda779c3c6c95a53e3b20af6386508946cd3"
Oct 31 14:02:23.444100 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:23.444076    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d07a417a529943aeef6e945444deda779c3c6c95a53e3b20af6386508946cd3\": container with ID starting with 6d07a417a529943aeef6e945444deda779c3c6c95a53e3b20af6386508946cd3 not found: ID does not exist" containerID="6d07a417a529943aeef6e945444deda779c3c6c95a53e3b20af6386508946cd3"
Oct 31 14:02:23.444147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.444112    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d07a417a529943aeef6e945444deda779c3c6c95a53e3b20af6386508946cd3"} err="failed to get container status \"6d07a417a529943aeef6e945444deda779c3c6c95a53e3b20af6386508946cd3\": rpc error: code = NotFound desc = could not find container \"6d07a417a529943aeef6e945444deda779c3c6c95a53e3b20af6386508946cd3\": container with ID starting with 6d07a417a529943aeef6e945444deda779c3c6c95a53e3b20af6386508946cd3 not found: ID does not exist"
Oct 31 14:02:23.824632 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:23.824550    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:24.260853 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.260808    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t" podUID="4ea944d0-8db8-4ae4-8e37-f04879e47a6a" containerName="payload-pre-processing" containerID="cri-o://c0d48616b67d2a1af035c4edfebc61fba790923710cf6fefa99141887c42d7aa" gracePeriod=25
Oct 31 14:02:24.397214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.397096    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:24.564361 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.564326    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:24.613306 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.613273    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"
Oct 31 14:02:24.647534 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.647499    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"
Oct 31 14:02:24.692677 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.692645    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"]
Oct 31 14:02:24.706408 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.706372    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"]
Oct 31 14:02:24.707829 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.707801    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-kube-api-access-4zv4r\" (UniqueName: \"kubernetes.io/projected/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-kube-api-access-4zv4r\") pod \"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6\" (UID: \"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6\") "
Oct 31 14:02:24.708032 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.708015    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-maas-api-xks-ca\") pod \"d664724b-e6a5-4cad-8317-b97fcfa2e0fe\" (UID: \"d664724b-e6a5-4cad-8317-b97fcfa2e0fe\") "
Oct 31 14:02:24.708130 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.708045    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-maas-api-xks-ca\") pod \"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6\" (UID: \"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6\") "
Oct 31 14:02:24.708492 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.708454    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-maas-api-openshift-ca\") pod \"d664724b-e6a5-4cad-8317-b97fcfa2e0fe\" (UID: \"d664724b-e6a5-4cad-8317-b97fcfa2e0fe\") "
Oct 31 14:02:24.708697 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.708665    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-maas-api-openshift-ca" pod "d664724b-e6a5-4cad-8317-b97fcfa2e0fe" (UID: "d664724b-e6a5-4cad-8317-b97fcfa2e0fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:24.709070 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.708551    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-kube-api-access-c7w52\" (UniqueName: \"kubernetes.io/projected/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-kube-api-access-c7w52\") pod \"d664724b-e6a5-4cad-8317-b97fcfa2e0fe\" (UID: \"d664724b-e6a5-4cad-8317-b97fcfa2e0fe\") "
Oct 31 14:02:24.709156 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.709128    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-maas-api-openshift-ca\") pod \"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6\" (UID: \"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6\") "
Oct 31 14:02:24.709594 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.709567    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-maas-api-openshift-ca" pod "4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6" (UID: "4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:24.709780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.709760    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:24.712198 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.712165    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-kube-api-access-4zv4r" pod "4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6" (UID: "4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6"). InnerVolumeSpecName "kube-api-access-4zv4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:24.717763 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.717732    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-maas-api-xks-ca" pod "4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6" (UID: "4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:24.718325 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.718303    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-kube-api-access-c7w52" pod "d664724b-e6a5-4cad-8317-b97fcfa2e0fe" (UID: "d664724b-e6a5-4cad-8317-b97fcfa2e0fe"). InnerVolumeSpecName "kube-api-access-c7w52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:24.728161 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.728128    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-maas-api-xks-ca" pod "d664724b-e6a5-4cad-8317-b97fcfa2e0fe" (UID: "d664724b-e6a5-4cad-8317-b97fcfa2e0fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:24.759038 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.759004    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eacbe93-226a-4def-96fe-b8409cc76f3b" path="/var/lib/kubelet/pods/2eacbe93-226a-4def-96fe-b8409cc76f3b/volumes"
Oct 31 14:02:24.759489 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.759471    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8922cd48-13c8-4172-9470-736ac36a46ed" path="/var/lib/kubelet/pods/8922cd48-13c8-4172-9470-736ac36a46ed/volumes"
Oct 31 14:02:24.811987 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.811889    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:24.811987 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.811926    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:24.811987 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.811944    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7w52\" (UniqueName: \"kubernetes.io/projected/d664724b-e6a5-4cad-8317-b97fcfa2e0fe-kube-api-access-c7w52\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:24.811987 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.811960    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:24.811987 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.811974    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zv4r\" (UniqueName: \"kubernetes.io/projected/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6-kube-api-access-4zv4r\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:24.827240 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.827184    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:24.847311 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.847263    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg" podUID="b042c863-4f10-4318-ac91-0464b31ebb89" containerName="payload-processing" containerID="cri-o://84ad66985cf7a6707d66b18f88efca9e49c2b28c522718d6a11eaebac3903cd4" gracePeriod=25
Oct 31 14:02:24.871054 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.871011    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28" podUID="ba6a83e5-80ac-4dca-bbce-fefc479e0abe" containerName="payload-pre-processing" containerID="cri-o://3fb7763817dab9c9b09d1206e9a8b662fb121be571b2e77e3e22f579179097be" gracePeriod=25
Oct 31 14:02:24.981757 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:24.981717    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:25.142508 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:25.142288    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:25.420208 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:25.419460    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"
Oct 31 14:02:25.420208 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:25.419467    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp" event={"ID":"4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6","Type":"ContainerDied","Data":"b5baf0672a4fb0efd292b1956d80bf974c2bbbd5a1f4751b441893f6cb0b6231"}
Oct 31 14:02:25.420208 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:25.419532    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5baf0672a4fb0efd292b1956d80bf974c2bbbd5a1f4751b441893f6cb0b6231"
Oct 31 14:02:25.426315 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:25.426255    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg" event={"ID":"d664724b-e6a5-4cad-8317-b97fcfa2e0fe","Type":"ContainerDied","Data":"39003101f24393c63a08436b33238d243e88fa559467c0e6cc105e274c86f275"}
Oct 31 14:02:25.426498 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:25.426323    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39003101f24393c63a08436b33238d243e88fa559467c0e6cc105e274c86f275"
Oct 31 14:02:25.426498 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:25.426361    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"
Oct 31 14:02:25.491353 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:25.481352    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:26.024230 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.024161    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 14:02:26.136692 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.136658    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9fb27a2-762a-4256-8c53-89a6bb50e654-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9fb27a2-762a-4256-8c53-89a6bb50e654-istiod-ca-cert\") pod \"b9fb27a2-762a-4256-8c53-89a6bb50e654\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") "
Oct 31 14:02:26.136963 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.136720    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-podinfo\") pod \"b9fb27a2-762a-4256-8c53-89a6bb50e654\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") "
Oct 31 14:02:26.136963 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.136746    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-data\") pod \"b9fb27a2-762a-4256-8c53-89a6bb50e654\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") "
Oct 31 14:02:26.136963 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.136821    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9fb27a2-762a-4256-8c53-89a6bb50e654-kube-api-access-f5g9w\" (UniqueName: \"kubernetes.io/projected/b9fb27a2-762a-4256-8c53-89a6bb50e654-kube-api-access-f5g9w\") pod \"b9fb27a2-762a-4256-8c53-89a6bb50e654\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") "
Oct 31 14:02:26.136963 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.136849    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-credential-socket\") pod \"b9fb27a2-762a-4256-8c53-89a6bb50e654\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") "
Oct 31 14:02:26.136963 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.136943    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-token\" (UniqueName: \"kubernetes.io/projected/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-token\") pod \"b9fb27a2-762a-4256-8c53-89a6bb50e654\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") "
Oct 31 14:02:26.137332 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.136975    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-envoy\") pod \"b9fb27a2-762a-4256-8c53-89a6bb50e654\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") "
Oct 31 14:02:26.137332 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.137004    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-workload-certs\") pod \"b9fb27a2-762a-4256-8c53-89a6bb50e654\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") "
Oct 31 14:02:26.137332 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.137089    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-workload-socket\") pod \"b9fb27a2-762a-4256-8c53-89a6bb50e654\" (UID: \"b9fb27a2-762a-4256-8c53-89a6bb50e654\") "
Oct 31 14:02:26.137332 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.137193    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-data" pod "b9fb27a2-762a-4256-8c53-89a6bb50e654" (UID: "b9fb27a2-762a-4256-8c53-89a6bb50e654"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:26.137332 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.137234    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9fb27a2-762a-4256-8c53-89a6bb50e654-istiod-ca-cert" pod "b9fb27a2-762a-4256-8c53-89a6bb50e654" (UID: "b9fb27a2-762a-4256-8c53-89a6bb50e654"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:26.137596 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.137455    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-workload-certs" pod "b9fb27a2-762a-4256-8c53-89a6bb50e654" (UID: "b9fb27a2-762a-4256-8c53-89a6bb50e654"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:26.137596 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.137499    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-credential-socket" pod "b9fb27a2-762a-4256-8c53-89a6bb50e654" (UID: "b9fb27a2-762a-4256-8c53-89a6bb50e654"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:26.137896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.137631    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-workload-socket" pod "b9fb27a2-762a-4256-8c53-89a6bb50e654" (UID: "b9fb27a2-762a-4256-8c53-89a6bb50e654"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:26.137896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.137735    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.137896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.137760    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.138079 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.137978    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.138079 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.138006    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9fb27a2-762a-4256-8c53-89a6bb50e654-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.139433 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.139405    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-envoy" pod "b9fb27a2-762a-4256-8c53-89a6bb50e654" (UID: "b9fb27a2-762a-4256-8c53-89a6bb50e654"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:26.139433 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.139424    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-podinfo" pod "b9fb27a2-762a-4256-8c53-89a6bb50e654" (UID: "b9fb27a2-762a-4256-8c53-89a6bb50e654"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:02:26.139819 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.139797    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-token" pod "b9fb27a2-762a-4256-8c53-89a6bb50e654" (UID: "b9fb27a2-762a-4256-8c53-89a6bb50e654"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:26.139819 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.139809    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fb27a2-762a-4256-8c53-89a6bb50e654-kube-api-access-f5g9w" pod "b9fb27a2-762a-4256-8c53-89a6bb50e654" (UID: "b9fb27a2-762a-4256-8c53-89a6bb50e654"). InnerVolumeSpecName "kube-api-access-f5g9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:26.191144 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.191100    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"]
Oct 31 14:02:26.239427 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.239376    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.239427 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.239426    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5g9w\" (UniqueName: \"kubernetes.io/projected/b9fb27a2-762a-4256-8c53-89a6bb50e654-kube-api-access-f5g9w\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.239652 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.239440    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.239652 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.239449    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.239652 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.239481    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9fb27a2-762a-4256-8c53-89a6bb50e654-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:26.443975 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.443842    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:26.445797 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.445773    4428 generic.go:356] "Generic (PLEG): container finished" podID="b9fb27a2-762a-4256-8c53-89a6bb50e654" containerID="9ffddf3cb81a87a3a21b322f62192d392287e6938ab5825ed15d2eae06344e3a" exitCode=0
Oct 31 14:02:26.445955 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.445825    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"
Oct 31 14:02:26.445955 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.445856    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n" event={"ID":"b9fb27a2-762a-4256-8c53-89a6bb50e654","Type":"ContainerDied","Data":"9ffddf3cb81a87a3a21b322f62192d392287e6938ab5825ed15d2eae06344e3a"}
Oct 31 14:02:26.446038 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.445966    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n" event={"ID":"b9fb27a2-762a-4256-8c53-89a6bb50e654","Type":"ContainerDied","Data":"99842e435721c810b37850cf12b390af15f306736013f26e9a74e930a78270ca"}
Oct 31 14:02:26.446038 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.445997    4428 scope.go:122] "RemoveContainer" containerID="9ffddf3cb81a87a3a21b322f62192d392287e6938ab5825ed15d2eae06344e3a"
Oct 31 14:02:26.457718 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.457696    4428 scope.go:122] "RemoveContainer" containerID="8b2d78d52433b2821008621c03483af3a2af47d779c95e2c2a80f872e1a77d0e"
Oct 31 14:02:26.467919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.467897    4428 scope.go:122] "RemoveContainer" containerID="9ffddf3cb81a87a3a21b322f62192d392287e6938ab5825ed15d2eae06344e3a"
Oct 31 14:02:26.468260 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:26.468236    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ffddf3cb81a87a3a21b322f62192d392287e6938ab5825ed15d2eae06344e3a\": container with ID starting with 9ffddf3cb81a87a3a21b322f62192d392287e6938ab5825ed15d2eae06344e3a not found: ID does not exist" containerID="9ffddf3cb81a87a3a21b322f62192d392287e6938ab5825ed15d2eae06344e3a"
Oct 31 14:02:26.468332 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.468269    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ffddf3cb81a87a3a21b322f62192d392287e6938ab5825ed15d2eae06344e3a"} err="failed to get container status \"9ffddf3cb81a87a3a21b322f62192d392287e6938ab5825ed15d2eae06344e3a\": rpc error: code = NotFound desc = could not find container \"9ffddf3cb81a87a3a21b322f62192d392287e6938ab5825ed15d2eae06344e3a\": container with ID starting with 9ffddf3cb81a87a3a21b322f62192d392287e6938ab5825ed15d2eae06344e3a not found: ID does not exist"
Oct 31 14:02:26.468332 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.468287    4428 scope.go:122] "RemoveContainer" containerID="8b2d78d52433b2821008621c03483af3a2af47d779c95e2c2a80f872e1a77d0e"
Oct 31 14:02:26.468563 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:26.468542    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2d78d52433b2821008621c03483af3a2af47d779c95e2c2a80f872e1a77d0e\": container with ID starting with 8b2d78d52433b2821008621c03483af3a2af47d779c95e2c2a80f872e1a77d0e not found: ID does not exist" containerID="8b2d78d52433b2821008621c03483af3a2af47d779c95e2c2a80f872e1a77d0e"
Oct 31 14:02:26.468613 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.468571    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2d78d52433b2821008621c03483af3a2af47d779c95e2c2a80f872e1a77d0e"} err="failed to get container status \"8b2d78d52433b2821008621c03483af3a2af47d779c95e2c2a80f872e1a77d0e\": rpc error: code = NotFound desc = could not find container \"8b2d78d52433b2821008621c03483af3a2af47d779c95e2c2a80f872e1a77d0e\": container with ID starting with 8b2d78d52433b2821008621c03483af3a2af47d779c95e2c2a80f872e1a77d0e not found: ID does not exist"
Oct 31 14:02:26.475605 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.475577    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"]
Oct 31 14:02:26.485954 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.485922    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-8zd7n"]
Oct 31 14:02:26.629797 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.629760    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:26.759397 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:26.759355    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9fb27a2-762a-4256-8c53-89a6bb50e654" path="/var/lib/kubelet/pods/b9fb27a2-762a-4256-8c53-89a6bb50e654/volumes"
Oct 31 14:02:27.300646 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:27.300586    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:28.095391 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:28.095354    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:28.895399 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:28.895369    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:29.062513 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.062466    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main" containerID="cri-o://d42338fd2971a02ebeff8dc4437b59667789764fc75047b524b9fc622d58594f" gracePeriod=45
Oct 31 14:02:29.338213 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.338182    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 14:02:29.363382 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.363347    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5" podUID="e8641866-a4d3-4c0d-8d8a-dd25e8572139" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:29.471844 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.471801    4428 generic.go:356] "Generic (PLEG): container finished" podID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerID="d42338fd2971a02ebeff8dc4437b59667789764fc75047b524b9fc622d58594f" exitCode=0
Oct 31 14:02:29.471844 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.471834    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"
Oct 31 14:02:29.472287 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.472064    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" event={"ID":"a85ed201-6d93-4c52-9315-8c733ea37b89","Type":"ContainerDied","Data":"d42338fd2971a02ebeff8dc4437b59667789764fc75047b524b9fc622d58594f"}
Oct 31 14:02:29.472287 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.472148    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v" event={"ID":"a85ed201-6d93-4c52-9315-8c733ea37b89","Type":"ContainerDied","Data":"bc10391b2b35dc5aa3e62d0b675a55c5fada486c0761723b5b10feb8281c60f3"}
Oct 31 14:02:29.472287 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.472182    4428 scope.go:122] "RemoveContainer" containerID="d42338fd2971a02ebeff8dc4437b59667789764fc75047b524b9fc622d58594f"
Oct 31 14:02:29.483774 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.483746    4428 scope.go:122] "RemoveContainer" containerID="d42338fd2971a02ebeff8dc4437b59667789764fc75047b524b9fc622d58594f"
Oct 31 14:02:29.484185 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:29.484155    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42338fd2971a02ebeff8dc4437b59667789764fc75047b524b9fc622d58594f\": container with ID starting with d42338fd2971a02ebeff8dc4437b59667789764fc75047b524b9fc622d58594f not found: ID does not exist" containerID="d42338fd2971a02ebeff8dc4437b59667789764fc75047b524b9fc622d58594f"
Oct 31 14:02:29.484333 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.484196    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42338fd2971a02ebeff8dc4437b59667789764fc75047b524b9fc622d58594f"} err="failed to get container status \"d42338fd2971a02ebeff8dc4437b59667789764fc75047b524b9fc622d58594f\": rpc error: code = NotFound desc = could not find container \"d42338fd2971a02ebeff8dc4437b59667789764fc75047b524b9fc622d58594f\": container with ID starting with d42338fd2971a02ebeff8dc4437b59667789764fc75047b524b9fc622d58594f not found: ID does not exist"
Oct 31 14:02:29.493211 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.493181    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-model-cache\") pod \"a85ed201-6d93-4c52-9315-8c733ea37b89\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") "
Oct 31 14:02:29.493382 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.493273    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-home\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-home\") pod \"a85ed201-6d93-4c52-9315-8c733ea37b89\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") "
Oct 31 14:02:29.493382 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.493301    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-tmp-dir\") pod \"a85ed201-6d93-4c52-9315-8c733ea37b89\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") "
Oct 31 14:02:29.493382 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.493359    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85ed201-6d93-4c52-9315-8c733ea37b89-kube-api-access-rlqpw\" (UniqueName: \"kubernetes.io/projected/a85ed201-6d93-4c52-9315-8c733ea37b89-kube-api-access-rlqpw\") pod \"a85ed201-6d93-4c52-9315-8c733ea37b89\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") "
Oct 31 14:02:29.493556 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.493412    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a85ed201-6d93-4c52-9315-8c733ea37b89-tls-certs\" (UniqueName: \"kubernetes.io/secret/a85ed201-6d93-4c52-9315-8c733ea37b89-tls-certs\") pod \"a85ed201-6d93-4c52-9315-8c733ea37b89\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") "
Oct 31 14:02:29.493556 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.493477    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-model-cache" pod "a85ed201-6d93-4c52-9315-8c733ea37b89" (UID: "a85ed201-6d93-4c52-9315-8c733ea37b89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:29.493665 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.493574    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-dshm\") pod \"a85ed201-6d93-4c52-9315-8c733ea37b89\" (UID: \"a85ed201-6d93-4c52-9315-8c733ea37b89\") "
Oct 31 14:02:29.493665 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.493586    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-home" pod "a85ed201-6d93-4c52-9315-8c733ea37b89" (UID: "a85ed201-6d93-4c52-9315-8c733ea37b89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:29.493665 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.493598    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-tmp-dir" pod "a85ed201-6d93-4c52-9315-8c733ea37b89" (UID: "a85ed201-6d93-4c52-9315-8c733ea37b89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:29.494085 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.494062    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:29.494085 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.494086    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:29.494436 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.494096    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:29.495955 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.495925    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85ed201-6d93-4c52-9315-8c733ea37b89-tls-certs" pod "a85ed201-6d93-4c52-9315-8c733ea37b89" (UID: "a85ed201-6d93-4c52-9315-8c733ea37b89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:29.495955 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.495950    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-dshm" pod "a85ed201-6d93-4c52-9315-8c733ea37b89" (UID: "a85ed201-6d93-4c52-9315-8c733ea37b89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:29.496287 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.496051    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85ed201-6d93-4c52-9315-8c733ea37b89-kube-api-access-rlqpw" pod "a85ed201-6d93-4c52-9315-8c733ea37b89" (UID: "a85ed201-6d93-4c52-9315-8c733ea37b89"). InnerVolumeSpecName "kube-api-access-rlqpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:29.595070 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.595036    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlqpw\" (UniqueName: \"kubernetes.io/projected/a85ed201-6d93-4c52-9315-8c733ea37b89-kube-api-access-rlqpw\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:29.595070 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.595068    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85ed201-6d93-4c52-9315-8c733ea37b89-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:29.595070 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.595080    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85ed201-6d93-4c52-9315-8c733ea37b89-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:29.693044 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.693002    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84" podUID="67e09833-8284-4dfa-a8f8-8f371ec49284" containerName="payload-processing" containerID="cri-o://5dedcfc490f11661731e45a0831b67e1590b18090021042063a468a8e8165a62" gracePeriod=25
Oct 31 14:02:29.706898 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.706850    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh" podUID="3a329323-cfeb-4213-950b-1ce6c8cc3fbe" containerName="payload-pre-processing" containerID="cri-o://fb941601b93953c2ce81404a15c1ec3c2625aaf4428252aa005a22c95ff33683" gracePeriod=25
Oct 31 14:02:29.811698 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.811587    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:29.819372 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.818604    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"]
Oct 31 14:02:29.823804 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.823779    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/distinct-2-w5-6784d6-kserve-8df4ccb5d-g8l8v"]
Oct 31 14:02:29.997423 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:29.997388    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"
Oct 31 14:02:30.059129 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.058858    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"
Oct 31 14:02:30.062212 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.062184    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"
Oct 31 14:02:30.103040 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.103004    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8641866-a4d3-4c0d-8d8a-dd25e8572139-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8641866-a4d3-4c0d-8d8a-dd25e8572139-plugins-config-volume\") pod \"e8641866-a4d3-4c0d-8d8a-dd25e8572139\" (UID: \"e8641866-a4d3-4c0d-8d8a-dd25e8572139\") "
Oct 31 14:02:30.103228 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.103215    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8641866-a4d3-4c0d-8d8a-dd25e8572139-kube-api-access-6zdmx\" (UniqueName: \"kubernetes.io/projected/e8641866-a4d3-4c0d-8d8a-dd25e8572139-kube-api-access-6zdmx\") pod \"e8641866-a4d3-4c0d-8d8a-dd25e8572139\" (UID: \"e8641866-a4d3-4c0d-8d8a-dd25e8572139\") "
Oct 31 14:02:30.103448 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.103420    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8641866-a4d3-4c0d-8d8a-dd25e8572139-plugins-config-volume" pod "e8641866-a4d3-4c0d-8d8a-dd25e8572139" (UID: "e8641866-a4d3-4c0d-8d8a-dd25e8572139"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:30.104255 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.104226    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8641866-a4d3-4c0d-8d8a-dd25e8572139-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:30.105773 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.105740    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8641866-a4d3-4c0d-8d8a-dd25e8572139-kube-api-access-6zdmx" pod "e8641866-a4d3-4c0d-8d8a-dd25e8572139" (UID: "e8641866-a4d3-4c0d-8d8a-dd25e8572139"). InnerVolumeSpecName "kube-api-access-6zdmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:30.204658 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.204619    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67e09833-8284-4dfa-a8f8-8f371ec49284-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e09833-8284-4dfa-a8f8-8f371ec49284-plugins-config-volume\") pod \"67e09833-8284-4dfa-a8f8-8f371ec49284\" (UID: \"67e09833-8284-4dfa-a8f8-8f371ec49284\") "
Oct 31 14:02:30.204858 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.204715    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e09833-8284-4dfa-a8f8-8f371ec49284-kube-api-access-22gj8\" (UniqueName: \"kubernetes.io/projected/67e09833-8284-4dfa-a8f8-8f371ec49284-kube-api-access-22gj8\") pod \"67e09833-8284-4dfa-a8f8-8f371ec49284\" (UID: \"67e09833-8284-4dfa-a8f8-8f371ec49284\") "
Oct 31 14:02:30.204858 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.204834    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a329323-cfeb-4213-950b-1ce6c8cc3fbe-kube-api-access-xwrq7\" (UniqueName: \"kubernetes.io/projected/3a329323-cfeb-4213-950b-1ce6c8cc3fbe-kube-api-access-xwrq7\") pod \"3a329323-cfeb-4213-950b-1ce6c8cc3fbe\" (UID: \"3a329323-cfeb-4213-950b-1ce6c8cc3fbe\") "
Oct 31 14:02:30.204995 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.204862    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a329323-cfeb-4213-950b-1ce6c8cc3fbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a329323-cfeb-4213-950b-1ce6c8cc3fbe-plugins-config-volume\") pod \"3a329323-cfeb-4213-950b-1ce6c8cc3fbe\" (UID: \"3a329323-cfeb-4213-950b-1ce6c8cc3fbe\") "
Oct 31 14:02:30.205278 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.205153    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e09833-8284-4dfa-a8f8-8f371ec49284-plugins-config-volume" pod "67e09833-8284-4dfa-a8f8-8f371ec49284" (UID: "67e09833-8284-4dfa-a8f8-8f371ec49284"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:30.205421 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.205320    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zdmx\" (UniqueName: \"kubernetes.io/projected/e8641866-a4d3-4c0d-8d8a-dd25e8572139-kube-api-access-6zdmx\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:30.205421 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.205335    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a329323-cfeb-4213-950b-1ce6c8cc3fbe-plugins-config-volume" pod "3a329323-cfeb-4213-950b-1ce6c8cc3fbe" (UID: "3a329323-cfeb-4213-950b-1ce6c8cc3fbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:30.207248 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.207224    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e09833-8284-4dfa-a8f8-8f371ec49284-kube-api-access-22gj8" pod "67e09833-8284-4dfa-a8f8-8f371ec49284" (UID: "67e09833-8284-4dfa-a8f8-8f371ec49284"). InnerVolumeSpecName "kube-api-access-22gj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:30.207443 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.207421    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a329323-cfeb-4213-950b-1ce6c8cc3fbe-kube-api-access-xwrq7" pod "3a329323-cfeb-4213-950b-1ce6c8cc3fbe" (UID: "3a329323-cfeb-4213-950b-1ce6c8cc3fbe"). InnerVolumeSpecName "kube-api-access-xwrq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:30.306938 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.306839    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwrq7\" (UniqueName: \"kubernetes.io/projected/3a329323-cfeb-4213-950b-1ce6c8cc3fbe-kube-api-access-xwrq7\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:30.306938 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.306909    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a329323-cfeb-4213-950b-1ce6c8cc3fbe-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:30.306938 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.306921    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e09833-8284-4dfa-a8f8-8f371ec49284-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:30.306938 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.306929    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22gj8\" (UniqueName: \"kubernetes.io/projected/67e09833-8284-4dfa-a8f8-8f371ec49284-kube-api-access-22gj8\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:30.481138 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.481104    4428 generic.go:356] "Generic (PLEG): container finished" podID="67e09833-8284-4dfa-a8f8-8f371ec49284" containerID="5dedcfc490f11661731e45a0831b67e1590b18090021042063a468a8e8165a62" exitCode=0
Oct 31 14:02:30.481332 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.481165    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"
Oct 31 14:02:30.481332 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.481199    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84" event={"ID":"67e09833-8284-4dfa-a8f8-8f371ec49284","Type":"ContainerDied","Data":"5dedcfc490f11661731e45a0831b67e1590b18090021042063a468a8e8165a62"}
Oct 31 14:02:30.481332 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.481292    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84" event={"ID":"67e09833-8284-4dfa-a8f8-8f371ec49284","Type":"ContainerDied","Data":"9a57fcec7d16e5ab2b9c68f8fbca45a747922bcc62c278034425a5f251fc8a00"}
Oct 31 14:02:30.481332 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.481326    4428 scope.go:122] "RemoveContainer" containerID="5dedcfc490f11661731e45a0831b67e1590b18090021042063a468a8e8165a62"
Oct 31 14:02:30.484288 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.484263    4428 generic.go:356] "Generic (PLEG): container finished" podID="e8641866-a4d3-4c0d-8d8a-dd25e8572139" containerID="0714db78f0b3bef9c995c8f3310afdf593508b279d76303f783d5246189760d5" exitCode=0
Oct 31 14:02:30.484442 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.484349    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"
Oct 31 14:02:30.484683 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.484343    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5" event={"ID":"e8641866-a4d3-4c0d-8d8a-dd25e8572139","Type":"ContainerDied","Data":"0714db78f0b3bef9c995c8f3310afdf593508b279d76303f783d5246189760d5"}
Oct 31 14:02:30.484770 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.484721    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5" event={"ID":"e8641866-a4d3-4c0d-8d8a-dd25e8572139","Type":"ContainerDied","Data":"e2e8822e98cc608d923e9258d37fc43769fa287834ebff8cbec959c377b3081c"}
Oct 31 14:02:30.486233 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.486207    4428 generic.go:356] "Generic (PLEG): container finished" podID="3a329323-cfeb-4213-950b-1ce6c8cc3fbe" containerID="fb941601b93953c2ce81404a15c1ec3c2625aaf4428252aa005a22c95ff33683" exitCode=0
Oct 31 14:02:30.486354 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.486298    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh" event={"ID":"3a329323-cfeb-4213-950b-1ce6c8cc3fbe","Type":"ContainerDied","Data":"fb941601b93953c2ce81404a15c1ec3c2625aaf4428252aa005a22c95ff33683"}
Oct 31 14:02:30.486354 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.486318    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"
Oct 31 14:02:30.486480 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.486376    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh" event={"ID":"3a329323-cfeb-4213-950b-1ce6c8cc3fbe","Type":"ContainerDied","Data":"daf6ca5f1eab4e03f3fab808a0d30fa7f5cfa0fb07da8f6b7f6a6f659ba51e1c"}
Oct 31 14:02:30.494984 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.494960    4428 scope.go:122] "RemoveContainer" containerID="5dedcfc490f11661731e45a0831b67e1590b18090021042063a468a8e8165a62"
Oct 31 14:02:30.495571 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:30.495540    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dedcfc490f11661731e45a0831b67e1590b18090021042063a468a8e8165a62\": container with ID starting with 5dedcfc490f11661731e45a0831b67e1590b18090021042063a468a8e8165a62 not found: ID does not exist" containerID="5dedcfc490f11661731e45a0831b67e1590b18090021042063a468a8e8165a62"
Oct 31 14:02:30.495727 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.495582    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dedcfc490f11661731e45a0831b67e1590b18090021042063a468a8e8165a62"} err="failed to get container status \"5dedcfc490f11661731e45a0831b67e1590b18090021042063a468a8e8165a62\": rpc error: code = NotFound desc = could not find container \"5dedcfc490f11661731e45a0831b67e1590b18090021042063a468a8e8165a62\": container with ID starting with 5dedcfc490f11661731e45a0831b67e1590b18090021042063a468a8e8165a62 not found: ID does not exist"
Oct 31 14:02:30.495727 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.495606    4428 scope.go:122] "RemoveContainer" containerID="0714db78f0b3bef9c995c8f3310afdf593508b279d76303f783d5246189760d5"
Oct 31 14:02:30.508293 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.508265    4428 scope.go:122] "RemoveContainer" containerID="0714db78f0b3bef9c995c8f3310afdf593508b279d76303f783d5246189760d5"
Oct 31 14:02:30.508706 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:30.508674    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0714db78f0b3bef9c995c8f3310afdf593508b279d76303f783d5246189760d5\": container with ID starting with 0714db78f0b3bef9c995c8f3310afdf593508b279d76303f783d5246189760d5 not found: ID does not exist" containerID="0714db78f0b3bef9c995c8f3310afdf593508b279d76303f783d5246189760d5"
Oct 31 14:02:30.508805 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.508716    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0714db78f0b3bef9c995c8f3310afdf593508b279d76303f783d5246189760d5"} err="failed to get container status \"0714db78f0b3bef9c995c8f3310afdf593508b279d76303f783d5246189760d5\": rpc error: code = NotFound desc = could not find container \"0714db78f0b3bef9c995c8f3310afdf593508b279d76303f783d5246189760d5\": container with ID starting with 0714db78f0b3bef9c995c8f3310afdf593508b279d76303f783d5246189760d5 not found: ID does not exist"
Oct 31 14:02:30.508805 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.508741    4428 scope.go:122] "RemoveContainer" containerID="fb941601b93953c2ce81404a15c1ec3c2625aaf4428252aa005a22c95ff33683"
Oct 31 14:02:30.521600 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.521580    4428 scope.go:122] "RemoveContainer" containerID="fb941601b93953c2ce81404a15c1ec3c2625aaf4428252aa005a22c95ff33683"
Oct 31 14:02:30.521711 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.521696    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"]
Oct 31 14:02:30.521994 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:30.521970    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb941601b93953c2ce81404a15c1ec3c2625aaf4428252aa005a22c95ff33683\": container with ID starting with fb941601b93953c2ce81404a15c1ec3c2625aaf4428252aa005a22c95ff33683 not found: ID does not exist" containerID="fb941601b93953c2ce81404a15c1ec3c2625aaf4428252aa005a22c95ff33683"
Oct 31 14:02:30.522072 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.522001    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb941601b93953c2ce81404a15c1ec3c2625aaf4428252aa005a22c95ff33683"} err="failed to get container status \"fb941601b93953c2ce81404a15c1ec3c2625aaf4428252aa005a22c95ff33683\": rpc error: code = NotFound desc = could not find container \"fb941601b93953c2ce81404a15c1ec3c2625aaf4428252aa005a22c95ff33683\": container with ID starting with fb941601b93953c2ce81404a15c1ec3c2625aaf4428252aa005a22c95ff33683 not found: ID does not exist"
Oct 31 14:02:30.524094 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.524069    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-fbd9bf65f-f5d84"]
Oct 31 14:02:30.536917 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.536888    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"]
Oct 31 14:02:30.542785 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.542759    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-7cb5494d4c-jnqnh"]
Oct 31 14:02:30.554468 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.554440    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"]
Oct 31 14:02:30.561151 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.561073    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-abdcb1-858c897c99-r6fx5"]
Oct 31 14:02:30.638902 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.634487    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:30.755929 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.755887    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a329323-cfeb-4213-950b-1ce6c8cc3fbe" path="/var/lib/kubelet/pods/3a329323-cfeb-4213-950b-1ce6c8cc3fbe/volumes"
Oct 31 14:02:30.756345 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.756331    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67e09833-8284-4dfa-a8f8-8f371ec49284" path="/var/lib/kubelet/pods/67e09833-8284-4dfa-a8f8-8f371ec49284/volumes"
Oct 31 14:02:30.756705 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.756693    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" path="/var/lib/kubelet/pods/a85ed201-6d93-4c52-9315-8c733ea37b89/volumes"
Oct 31 14:02:30.757375 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.757359    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8641866-a4d3-4c0d-8d8a-dd25e8572139" path="/var/lib/kubelet/pods/e8641866-a4d3-4c0d-8d8a-dd25e8572139/volumes"
Oct 31 14:02:30.841065 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:30.840974    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg" podUID="b042c863-4f10-4318-ac91-0464b31ebb89" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:31.034023 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.033986    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t" podUID="4ea944d0-8db8-4ae4-8e37-f04879e47a6a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:31.193567 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.193480    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:31.305786 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.305756    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"]
Oct 31 14:02:31.306359 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306344    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b9fb27a2-762a-4256-8c53-89a6bb50e654" containerName="istio-proxy"
Oct 31 14:02:31.306406 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306361    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9fb27a2-762a-4256-8c53-89a6bb50e654" containerName="istio-proxy"
Oct 31 14:02:31.306406 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306377    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b9fb27a2-762a-4256-8c53-89a6bb50e654" containerName="istio-proxy"
Oct 31 14:02:31.306406 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306382    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9fb27a2-762a-4256-8c53-89a6bb50e654" containerName="istio-proxy"
Oct 31 14:02:31.306406 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306394    4428 cpu_manager.go:414] "containerMap: removing container" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main"
Oct 31 14:02:31.306406 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306399    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main"
Oct 31 14:02:31.306575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306408    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d664724b-e6a5-4cad-8317-b97fcfa2e0fe" containerName="revoke-keys"
Oct 31 14:02:31.306575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306413    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d664724b-e6a5-4cad-8317-b97fcfa2e0fe" containerName="revoke-keys"
Oct 31 14:02:31.306575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306421    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e8641866-a4d3-4c0d-8d8a-dd25e8572139" containerName="payload-processing"
Oct 31 14:02:31.306575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306426    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8641866-a4d3-4c0d-8d8a-dd25e8572139" containerName="payload-processing"
Oct 31 14:02:31.306575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306434    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6" containerName="revoke-keys"
Oct 31 14:02:31.306575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306438    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6" containerName="revoke-keys"
Oct 31 14:02:31.306575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306449    4428 cpu_manager.go:414] "containerMap: removing container" podUID="67e09833-8284-4dfa-a8f8-8f371ec49284" containerName="payload-processing"
Oct 31 14:02:31.306575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306453    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e09833-8284-4dfa-a8f8-8f371ec49284" containerName="payload-processing"
Oct 31 14:02:31.306575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306474    4428 cpu_manager.go:414] "containerMap: removing container" podUID="2eacbe93-226a-4def-96fe-b8409cc76f3b" containerName="maas-api"
Oct 31 14:02:31.306575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306479    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eacbe93-226a-4def-96fe-b8409cc76f3b" containerName="maas-api"
Oct 31 14:02:31.306575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306486    4428 cpu_manager.go:414] "containerMap: removing container" podUID="3a329323-cfeb-4213-950b-1ce6c8cc3fbe" containerName="payload-pre-processing"
Oct 31 14:02:31.306575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306491    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a329323-cfeb-4213-950b-1ce6c8cc3fbe" containerName="payload-pre-processing"
Oct 31 14:02:31.306575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306563    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e09833-8284-4dfa-a8f8-8f371ec49284" containerName="payload-processing"
Oct 31 14:02:31.306575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306575    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eacbe93-226a-4def-96fe-b8409cc76f3b" containerName="maas-api"
Oct 31 14:02:31.306575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306581    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a329323-cfeb-4213-950b-1ce6c8cc3fbe" containerName="payload-pre-processing"
Oct 31 14:02:31.307051 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306589    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9fb27a2-762a-4256-8c53-89a6bb50e654" containerName="istio-proxy"
Oct 31 14:02:31.307051 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306594    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85ed201-6d93-4c52-9315-8c733ea37b89" containerName="main"
Oct 31 14:02:31.307051 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306599    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d664724b-e6a5-4cad-8317-b97fcfa2e0fe" containerName="revoke-keys"
Oct 31 14:02:31.307051 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306605    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8641866-a4d3-4c0d-8d8a-dd25e8572139" containerName="payload-processing"
Oct 31 14:02:31.307051 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.306611    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6" containerName="revoke-keys"
Oct 31 14:02:31.311705 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.311688    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"
Oct 31 14:02:31.323606 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.323576    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"]
Oct 31 14:02:31.366937 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.364035    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1b75f9\""
Oct 31 14:02:31.366937 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.364187    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1b75f9-dockercfg-4tp2n\""
Oct 31 14:02:31.392903 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.389558    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:31.440656 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.440607    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a646d3-fa24-4bde-a717-14d3d3ff4c4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv\" (UID: \"58a646d3-fa24-4bde-a717-14d3d3ff4c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"
Oct 31 14:02:31.440958 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.440712    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnr5n\" (UniqueName: \"kubernetes.io/projected/58a646d3-fa24-4bde-a717-14d3d3ff4c4e-kube-api-access-qnr5n\") pod \"payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv\" (UID: \"58a646d3-fa24-4bde-a717-14d3d3ff4c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"
Oct 31 14:02:31.451340 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.451240    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"]
Oct 31 14:02:31.542105 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.542074    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a646d3-fa24-4bde-a717-14d3d3ff4c4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv\" (UID: \"58a646d3-fa24-4bde-a717-14d3d3ff4c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"
Oct 31 14:02:31.542305 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.542143    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnr5n\" (UniqueName: \"kubernetes.io/projected/58a646d3-fa24-4bde-a717-14d3d3ff4c4e-kube-api-access-qnr5n\") pod \"payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv\" (UID: \"58a646d3-fa24-4bde-a717-14d3d3ff4c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"
Oct 31 14:02:31.542844 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.542823    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a646d3-fa24-4bde-a717-14d3d3ff4c4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv\" (UID: \"58a646d3-fa24-4bde-a717-14d3d3ff4c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"
Oct 31 14:02:31.564400 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.564367    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnr5n\" (UniqueName: \"kubernetes.io/projected/58a646d3-fa24-4bde-a717-14d3d3ff4c4e-kube-api-access-qnr5n\") pod \"payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv\" (UID: \"58a646d3-fa24-4bde-a717-14d3d3ff4c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"
Oct 31 14:02:31.600132 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.599984    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"]
Oct 31 14:02:31.600326 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.600281    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb" podUID="b290805f-2d02-4d45-abbd-d0f0f8afe535" containerName="istio-proxy" containerID="cri-o://9b68c96b43dc262c1dd85a9d0c4032fbc64d8b083dfd3033a56d13039f9dba34" gracePeriod=30
Oct 31 14:02:31.603950 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.603827    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:31.627438 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.627411    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"
Oct 31 14:02:31.812655 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.809045    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:31.855234 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.855200    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28" podUID="ba6a83e5-80ac-4dca-bbce-fefc479e0abe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:31.921713 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.921683    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"]
Oct 31 14:02:31.923844 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:02:31.923811    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a646d3_fa24_4bde_a717_14d3d3ff4c4e.slice/crio-47aea849db024281c82328462b1065670f6d577e8ed54a7f429df9bb33583542 WatchSource:0}: Error finding container 47aea849db024281c82328462b1065670f6d577e8ed54a7f429df9bb33583542: Status 404 returned error can't find the container with id 47aea849db024281c82328462b1065670f6d577e8ed54a7f429df9bb33583542
Oct 31 14:02:31.993007 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:31.992970    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:32.153225 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.153128    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:32.271718 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.271679    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"]
Oct 31 14:02:32.272712 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.272693    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9fb27a2-762a-4256-8c53-89a6bb50e654" containerName="istio-proxy"
Oct 31 14:02:32.276678 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.276655    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"
Oct 31 14:02:32.294732 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.294061    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:02:32.294732 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.294109    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:02:32.299429 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.299402    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"]
Oct 31 14:02:32.379504 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.379466    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2mqf\" (UniqueName: \"kubernetes.io/projected/bd6dba80-2e73-4b66-809a-3383e52fd9fb-kube-api-access-b2mqf\") pod \"maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq\" (UID: \"bd6dba80-2e73-4b66-809a-3383e52fd9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"
Oct 31 14:02:32.379680 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.379619    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6dba80-2e73-4b66-809a-3383e52fd9fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq\" (UID: \"bd6dba80-2e73-4b66-809a-3383e52fd9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"
Oct 31 14:02:32.379749 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.379703    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6dba80-2e73-4b66-809a-3383e52fd9fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq\" (UID: \"bd6dba80-2e73-4b66-809a-3383e52fd9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"
Oct 31 14:02:32.481480 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.481447    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6dba80-2e73-4b66-809a-3383e52fd9fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq\" (UID: \"bd6dba80-2e73-4b66-809a-3383e52fd9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"
Oct 31 14:02:32.481706 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.481523    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6dba80-2e73-4b66-809a-3383e52fd9fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq\" (UID: \"bd6dba80-2e73-4b66-809a-3383e52fd9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"
Oct 31 14:02:32.481706 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.481610    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2mqf\" (UniqueName: \"kubernetes.io/projected/bd6dba80-2e73-4b66-809a-3383e52fd9fb-kube-api-access-b2mqf\") pod \"maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq\" (UID: \"bd6dba80-2e73-4b66-809a-3383e52fd9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"
Oct 31 14:02:32.482225 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.482196    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6dba80-2e73-4b66-809a-3383e52fd9fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq\" (UID: \"bd6dba80-2e73-4b66-809a-3383e52fd9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"
Oct 31 14:02:32.484153 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.484135    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6dba80-2e73-4b66-809a-3383e52fd9fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq\" (UID: \"bd6dba80-2e73-4b66-809a-3383e52fd9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"
Oct 31 14:02:32.495599 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.495575    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2mqf\" (UniqueName: \"kubernetes.io/projected/bd6dba80-2e73-4b66-809a-3383e52fd9fb-kube-api-access-b2mqf\") pod \"maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq\" (UID: \"bd6dba80-2e73-4b66-809a-3383e52fd9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"
Oct 31 14:02:32.509468 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.509425    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv" event={"ID":"58a646d3-fa24-4bde-a717-14d3d3ff4c4e","Type":"ContainerStarted","Data":"9c305ee29030099c2830b3aba351d9195d508734f4c98f591eda4ce6280feb7a"}
Oct 31 14:02:32.509600 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.509488    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv" event={"ID":"58a646d3-fa24-4bde-a717-14d3d3ff4c4e","Type":"ContainerStarted","Data":"47aea849db024281c82328462b1065670f6d577e8ed54a7f429df9bb33583542"}
Oct 31 14:02:32.509600 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.509517    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"
Oct 31 14:02:32.533660 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.533606    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv" podStartSLOduration=1.53359056 podStartE2EDuration="1.53359056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:32.530475114 +0000 UTC m=+3549.928823740" watchObservedRunningTime="2026-10-31 14:02:32.53359056 +0000 UTC m=+3549.931939184"
Oct 31 14:02:32.597354 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.597324    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"
Oct 31 14:02:32.916147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:32.916116    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"]
Oct 31 14:02:32.918039 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:02:32.918011    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6dba80_2e73_4b66_809a_3383e52fd9fb.slice/crio-18372935454cb58ed7e43352ad46b59ef938eb2733346ee27f532160b6bf51c0 WatchSource:0}: Error finding container 18372935454cb58ed7e43352ad46b59ef938eb2733346ee27f532160b6bf51c0: Status 404 returned error can't find the container with id 18372935454cb58ed7e43352ad46b59ef938eb2733346ee27f532160b6bf51c0
Oct 31 14:02:33.142792 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:33.142755    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:33.528157 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:33.528012    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq" event={"ID":"bd6dba80-2e73-4b66-809a-3383e52fd9fb","Type":"ContainerStarted","Data":"14626cc75a43f4e592fde67eabaefd0e3ec029dda77b8645dd16cbdbf08fefbb"}
Oct 31 14:02:33.528157 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:33.528125    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq" event={"ID":"bd6dba80-2e73-4b66-809a-3383e52fd9fb","Type":"ContainerStarted","Data":"18372935454cb58ed7e43352ad46b59ef938eb2733346ee27f532160b6bf51c0"}
Oct 31 14:02:33.566174 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:33.566117    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq" podStartSLOduration=1.566102285 podStartE2EDuration="1.566102285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:33.56380792 +0000 UTC m=+3550.962156543" watchObservedRunningTime="2026-10-31 14:02:33.566102285 +0000 UTC m=+3550.964450909"
Oct 31 14:02:34.103249 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:34.103104    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:34.536760 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:34.536727    4428 generic.go:356] "Generic (PLEG): container finished" podID="bd6dba80-2e73-4b66-809a-3383e52fd9fb" containerID="14626cc75a43f4e592fde67eabaefd0e3ec029dda77b8645dd16cbdbf08fefbb" exitCode=0
Oct 31 14:02:34.537288 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:34.536809    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq" event={"ID":"bd6dba80-2e73-4b66-809a-3383e52fd9fb","Type":"ContainerDied","Data":"14626cc75a43f4e592fde67eabaefd0e3ec029dda77b8645dd16cbdbf08fefbb"}
Oct 31 14:02:34.964233 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:34.964194    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:35.747032 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:35.746829    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:35.758249 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:35.758218    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"
Oct 31 14:02:35.938849 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:35.938758    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6dba80-2e73-4b66-809a-3383e52fd9fb-kube-api-access-b2mqf\" (UniqueName: \"kubernetes.io/projected/bd6dba80-2e73-4b66-809a-3383e52fd9fb-kube-api-access-b2mqf\") pod \"bd6dba80-2e73-4b66-809a-3383e52fd9fb\" (UID: \"bd6dba80-2e73-4b66-809a-3383e52fd9fb\") "
Oct 31 14:02:35.939051 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:35.938957    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd6dba80-2e73-4b66-809a-3383e52fd9fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6dba80-2e73-4b66-809a-3383e52fd9fb-maas-api-xks-ca\") pod \"bd6dba80-2e73-4b66-809a-3383e52fd9fb\" (UID: \"bd6dba80-2e73-4b66-809a-3383e52fd9fb\") "
Oct 31 14:02:35.939051 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:35.939000    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd6dba80-2e73-4b66-809a-3383e52fd9fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6dba80-2e73-4b66-809a-3383e52fd9fb-maas-api-openshift-ca\") pod \"bd6dba80-2e73-4b66-809a-3383e52fd9fb\" (UID: \"bd6dba80-2e73-4b66-809a-3383e52fd9fb\") "
Oct 31 14:02:35.939418 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:35.939393    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6dba80-2e73-4b66-809a-3383e52fd9fb-maas-api-openshift-ca" pod "bd6dba80-2e73-4b66-809a-3383e52fd9fb" (UID: "bd6dba80-2e73-4b66-809a-3383e52fd9fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:35.941172 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:35.941151    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6dba80-2e73-4b66-809a-3383e52fd9fb-kube-api-access-b2mqf" pod "bd6dba80-2e73-4b66-809a-3383e52fd9fb" (UID: "bd6dba80-2e73-4b66-809a-3383e52fd9fb"). InnerVolumeSpecName "kube-api-access-b2mqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:35.941272 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:35.941237    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd6dba80-2e73-4b66-809a-3383e52fd9fb-maas-api-xks-ca" pod "bd6dba80-2e73-4b66-809a-3383e52fd9fb" (UID: "bd6dba80-2e73-4b66-809a-3383e52fd9fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:36.040065 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:36.040020    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2mqf\" (UniqueName: \"kubernetes.io/projected/bd6dba80-2e73-4b66-809a-3383e52fd9fb-kube-api-access-b2mqf\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:36.040065 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:36.040049    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6dba80-2e73-4b66-809a-3383e52fd9fb-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:36.040065 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:36.040071    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6dba80-2e73-4b66-809a-3383e52fd9fb-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:36.451966 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:36.451919    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb" podUID="ee893209-7839-4a39-8aaa-17dc0f90d8d7" containerName="payload-pre-processing" containerID="cri-o://bc0ed2137f062dbff31cbab5f0d1f1804312409d8a3ed243ae582fa39c7b2e44" gracePeriod=25
Oct 31 14:02:36.553113 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:36.553085    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"
Oct 31 14:02:36.553276 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:36.553111    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq" event={"ID":"bd6dba80-2e73-4b66-809a-3383e52fd9fb","Type":"ContainerDied","Data":"18372935454cb58ed7e43352ad46b59ef938eb2733346ee27f532160b6bf51c0"}
Oct 31 14:02:36.553276 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:36.553180    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18372935454cb58ed7e43352ad46b59ef938eb2733346ee27f532160b6bf51c0"
Oct 31 14:02:36.610935 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:36.610883    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:36.877009 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:36.876975    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 14:02:37.052295 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.052206    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-podinfo\") pod \"b290805f-2d02-4d45-abbd-d0f0f8afe535\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") "
Oct 31 14:02:37.052490 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.052303    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-envoy\") pod \"b290805f-2d02-4d45-abbd-d0f0f8afe535\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") "
Oct 31 14:02:37.052490 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.052337    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-data\") pod \"b290805f-2d02-4d45-abbd-d0f0f8afe535\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") "
Oct 31 14:02:37.052490 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.052404    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-workload-socket\") pod \"b290805f-2d02-4d45-abbd-d0f0f8afe535\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") "
Oct 31 14:02:37.052490 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.052424    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b290805f-2d02-4d45-abbd-d0f0f8afe535-kube-api-access-wf2ld\" (UniqueName: \"kubernetes.io/projected/b290805f-2d02-4d45-abbd-d0f0f8afe535-kube-api-access-wf2ld\") pod \"b290805f-2d02-4d45-abbd-d0f0f8afe535\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") "
Oct 31 14:02:37.052490 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.052448    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b290805f-2d02-4d45-abbd-d0f0f8afe535-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b290805f-2d02-4d45-abbd-d0f0f8afe535-istiod-ca-cert\") pod \"b290805f-2d02-4d45-abbd-d0f0f8afe535\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") "
Oct 31 14:02:37.052490 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.052471    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-workload-certs\") pod \"b290805f-2d02-4d45-abbd-d0f0f8afe535\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") "
Oct 31 14:02:37.052841 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.052508    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-credential-socket\") pod \"b290805f-2d02-4d45-abbd-d0f0f8afe535\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") "
Oct 31 14:02:37.052841 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.052544    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-token\" (UniqueName: \"kubernetes.io/projected/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-token\") pod \"b290805f-2d02-4d45-abbd-d0f0f8afe535\" (UID: \"b290805f-2d02-4d45-abbd-d0f0f8afe535\") "
Oct 31 14:02:37.052841 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.052797    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-workload-socket" pod "b290805f-2d02-4d45-abbd-d0f0f8afe535" (UID: "b290805f-2d02-4d45-abbd-d0f0f8afe535"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:37.053078 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.052946    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-workload-certs" pod "b290805f-2d02-4d45-abbd-d0f0f8afe535" (UID: "b290805f-2d02-4d45-abbd-d0f0f8afe535"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:37.053078 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.053048    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-credential-socket" pod "b290805f-2d02-4d45-abbd-d0f0f8afe535" (UID: "b290805f-2d02-4d45-abbd-d0f0f8afe535"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:37.053393 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.053365    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.053468 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.053399    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.053468 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.053415    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.053671 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.053637    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b290805f-2d02-4d45-abbd-d0f0f8afe535-istiod-ca-cert" pod "b290805f-2d02-4d45-abbd-d0f0f8afe535" (UID: "b290805f-2d02-4d45-abbd-d0f0f8afe535"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:37.053857 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.053834    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-data" pod "b290805f-2d02-4d45-abbd-d0f0f8afe535" (UID: "b290805f-2d02-4d45-abbd-d0f0f8afe535"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:37.055115 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.055090    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-envoy" pod "b290805f-2d02-4d45-abbd-d0f0f8afe535" (UID: "b290805f-2d02-4d45-abbd-d0f0f8afe535"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:37.055319 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.055294    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-podinfo" pod "b290805f-2d02-4d45-abbd-d0f0f8afe535" (UID: "b290805f-2d02-4d45-abbd-d0f0f8afe535"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:02:37.055440 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.055416    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b290805f-2d02-4d45-abbd-d0f0f8afe535-kube-api-access-wf2ld" pod "b290805f-2d02-4d45-abbd-d0f0f8afe535" (UID: "b290805f-2d02-4d45-abbd-d0f0f8afe535"). InnerVolumeSpecName "kube-api-access-wf2ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:37.055572 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.055553    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-token" pod "b290805f-2d02-4d45-abbd-d0f0f8afe535" (UID: "b290805f-2d02-4d45-abbd-d0f0f8afe535"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:37.144671 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.144636    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"]
Oct 31 14:02:37.154598 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.154567    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.154598 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.154593    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf2ld\" (UniqueName: \"kubernetes.io/projected/b290805f-2d02-4d45-abbd-d0f0f8afe535-kube-api-access-wf2ld\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.154598 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.154604    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b290805f-2d02-4d45-abbd-d0f0f8afe535-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.154940 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.154613    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.154940 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.154621    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.154940 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.154630    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b290805f-2d02-4d45-abbd-d0f0f8afe535-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:37.163203 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.163172    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"]
Oct 31 14:02:37.316532 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.316444    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:37.481365 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.481323    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:37.520639 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.520589    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"]
Oct 31 14:02:37.521255 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.521238    4428 cpu_manager.go:414] "containerMap: removing container" podUID="bd6dba80-2e73-4b66-809a-3383e52fd9fb" containerName="revoke-keys"
Oct 31 14:02:37.521303 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.521257    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6dba80-2e73-4b66-809a-3383e52fd9fb" containerName="revoke-keys"
Oct 31 14:02:37.521303 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.521288    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b290805f-2d02-4d45-abbd-d0f0f8afe535" containerName="istio-proxy"
Oct 31 14:02:37.521303 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.521294    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b290805f-2d02-4d45-abbd-d0f0f8afe535" containerName="istio-proxy"
Oct 31 14:02:37.521303 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.521301    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b290805f-2d02-4d45-abbd-d0f0f8afe535" containerName="istio-proxy"
Oct 31 14:02:37.521455 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.521305    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b290805f-2d02-4d45-abbd-d0f0f8afe535" containerName="istio-proxy"
Oct 31 14:02:37.521455 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.521396    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6dba80-2e73-4b66-809a-3383e52fd9fb" containerName="revoke-keys"
Oct 31 14:02:37.521455 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.521410    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b290805f-2d02-4d45-abbd-d0f0f8afe535" containerName="istio-proxy"
Oct 31 14:02:37.521455 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.521416    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b290805f-2d02-4d45-abbd-d0f0f8afe535" containerName="istio-proxy"
Oct 31 14:02:37.525618 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.525602    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"
Oct 31 14:02:37.532474 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.532423    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:02:37.532696 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.532649    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:02:37.535238 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.534586    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"]
Oct 31 14:02:37.543925 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.543898    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"]
Oct 31 14:02:37.544094 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.543965    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"
Oct 31 14:02:37.574115 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.572919    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"]
Oct 31 14:02:37.640467 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.640431    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24885cc-c332-444e-89ba-9232a2b43196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5028dced-8wprp\" (UID: \"b24885cc-c332-444e-89ba-9232a2b43196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"
Oct 31 14:02:37.640697 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.640497    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3bb47f6-qvsvt\" (UID: \"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"
Oct 31 14:02:37.640697 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.640629    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdqgs\" (UniqueName: \"kubernetes.io/projected/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-kube-api-access-tdqgs\") pod \"maas-api-revoke-sub-a3bb47f6-qvsvt\" (UID: \"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"
Oct 31 14:02:37.640697 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.640668    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24885cc-c332-444e-89ba-9232a2b43196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5028dced-8wprp\" (UID: \"b24885cc-c332-444e-89ba-9232a2b43196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"
Oct 31 14:02:37.640929 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.640735    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3bb47f6-qvsvt\" (UID: \"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"
Oct 31 14:02:37.640929 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.640771    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x57qr\" (UniqueName: \"kubernetes.io/projected/b24885cc-c332-444e-89ba-9232a2b43196-kube-api-access-x57qr\") pod \"maas-api-revoke-sub-5028dced-8wprp\" (UID: \"b24885cc-c332-444e-89ba-9232a2b43196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"
Oct 31 14:02:37.673193 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.673152    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:37.673551 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.673519    4428 generic.go:356] "Generic (PLEG): container finished" podID="b290805f-2d02-4d45-abbd-d0f0f8afe535" containerID="9b68c96b43dc262c1dd85a9d0c4032fbc64d8b083dfd3033a56d13039f9dba34" exitCode=0
Oct 31 14:02:37.673832 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.673811    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb" event={"ID":"b290805f-2d02-4d45-abbd-d0f0f8afe535","Type":"ContainerDied","Data":"9b68c96b43dc262c1dd85a9d0c4032fbc64d8b083dfd3033a56d13039f9dba34"}
Oct 31 14:02:37.673982 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.673964    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb" event={"ID":"b290805f-2d02-4d45-abbd-d0f0f8afe535","Type":"ContainerDied","Data":"80e8ec983302e5855efe30f113ef19f25b71840e916ec16da77908b62a2ad4cc"}
Oct 31 14:02:37.674049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.674006    4428 scope.go:122] "RemoveContainer" containerID="9b68c96b43dc262c1dd85a9d0c4032fbc64d8b083dfd3033a56d13039f9dba34"
Oct 31 14:02:37.674375 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.674224    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"
Oct 31 14:02:37.686090 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.686062    4428 scope.go:122] "RemoveContainer" containerID="2e621b67784052dd20cecfbcbb2092d3c74843eae22e2a62a4f98e6b7f109989"
Oct 31 14:02:37.695934 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.695860    4428 scope.go:122] "RemoveContainer" containerID="9b68c96b43dc262c1dd85a9d0c4032fbc64d8b083dfd3033a56d13039f9dba34"
Oct 31 14:02:37.696302 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:37.696278    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b68c96b43dc262c1dd85a9d0c4032fbc64d8b083dfd3033a56d13039f9dba34\": container with ID starting with 9b68c96b43dc262c1dd85a9d0c4032fbc64d8b083dfd3033a56d13039f9dba34 not found: ID does not exist" containerID="9b68c96b43dc262c1dd85a9d0c4032fbc64d8b083dfd3033a56d13039f9dba34"
Oct 31 14:02:37.696394 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.696311    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b68c96b43dc262c1dd85a9d0c4032fbc64d8b083dfd3033a56d13039f9dba34"} err="failed to get container status \"9b68c96b43dc262c1dd85a9d0c4032fbc64d8b083dfd3033a56d13039f9dba34\": rpc error: code = NotFound desc = could not find container \"9b68c96b43dc262c1dd85a9d0c4032fbc64d8b083dfd3033a56d13039f9dba34\": container with ID starting with 9b68c96b43dc262c1dd85a9d0c4032fbc64d8b083dfd3033a56d13039f9dba34 not found: ID does not exist"
Oct 31 14:02:37.696394 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.696331    4428 scope.go:122] "RemoveContainer" containerID="2e621b67784052dd20cecfbcbb2092d3c74843eae22e2a62a4f98e6b7f109989"
Oct 31 14:02:37.696614 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:37.696594    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e621b67784052dd20cecfbcbb2092d3c74843eae22e2a62a4f98e6b7f109989\": container with ID starting with 2e621b67784052dd20cecfbcbb2092d3c74843eae22e2a62a4f98e6b7f109989 not found: ID does not exist" containerID="2e621b67784052dd20cecfbcbb2092d3c74843eae22e2a62a4f98e6b7f109989"
Oct 31 14:02:37.696660 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.696623    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e621b67784052dd20cecfbcbb2092d3c74843eae22e2a62a4f98e6b7f109989"} err="failed to get container status \"2e621b67784052dd20cecfbcbb2092d3c74843eae22e2a62a4f98e6b7f109989\": rpc error: code = NotFound desc = could not find container \"2e621b67784052dd20cecfbcbb2092d3c74843eae22e2a62a4f98e6b7f109989\": container with ID starting with 2e621b67784052dd20cecfbcbb2092d3c74843eae22e2a62a4f98e6b7f109989 not found: ID does not exist"
Oct 31 14:02:37.704306 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.704278    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"]
Oct 31 14:02:37.712366 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.712336    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5e8f29-openshift-default-745c6666c9-fkrpb"]
Oct 31 14:02:37.741632 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.741595    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x57qr\" (UniqueName: \"kubernetes.io/projected/b24885cc-c332-444e-89ba-9232a2b43196-kube-api-access-x57qr\") pod \"maas-api-revoke-sub-5028dced-8wprp\" (UID: \"b24885cc-c332-444e-89ba-9232a2b43196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"
Oct 31 14:02:37.741805 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.741649    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24885cc-c332-444e-89ba-9232a2b43196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5028dced-8wprp\" (UID: \"b24885cc-c332-444e-89ba-9232a2b43196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"
Oct 31 14:02:37.741805 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.741678    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3bb47f6-qvsvt\" (UID: \"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"
Oct 31 14:02:37.741805 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.741755    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdqgs\" (UniqueName: \"kubernetes.io/projected/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-kube-api-access-tdqgs\") pod \"maas-api-revoke-sub-a3bb47f6-qvsvt\" (UID: \"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"
Oct 31 14:02:37.741805 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.741785    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24885cc-c332-444e-89ba-9232a2b43196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5028dced-8wprp\" (UID: \"b24885cc-c332-444e-89ba-9232a2b43196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"
Oct 31 14:02:37.742073 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.741834    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3bb47f6-qvsvt\" (UID: \"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"
Oct 31 14:02:37.742409 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.742378    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24885cc-c332-444e-89ba-9232a2b43196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5028dced-8wprp\" (UID: \"b24885cc-c332-444e-89ba-9232a2b43196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"
Oct 31 14:02:37.742409 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.742395    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3bb47f6-qvsvt\" (UID: \"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"
Oct 31 14:02:37.744396 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.744373    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24885cc-c332-444e-89ba-9232a2b43196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5028dced-8wprp\" (UID: \"b24885cc-c332-444e-89ba-9232a2b43196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"
Oct 31 14:02:37.744490 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.744433    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3bb47f6-qvsvt\" (UID: \"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"
Oct 31 14:02:37.754675 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.754635    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x57qr\" (UniqueName: \"kubernetes.io/projected/b24885cc-c332-444e-89ba-9232a2b43196-kube-api-access-x57qr\") pod \"maas-api-revoke-sub-5028dced-8wprp\" (UID: \"b24885cc-c332-444e-89ba-9232a2b43196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"
Oct 31 14:02:37.754835 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.754733    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdqgs\" (UniqueName: \"kubernetes.io/projected/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-kube-api-access-tdqgs\") pod \"maas-api-revoke-sub-a3bb47f6-qvsvt\" (UID: \"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"
Oct 31 14:02:37.846733 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.846623    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"
Oct 31 14:02:37.863621 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.863593    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:37.973496 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:37.973456    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"
Oct 31 14:02:38.150352 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:38.138672    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"]
Oct 31 14:02:38.176477 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:38.176446    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:38.277065 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:38.277023    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"]
Oct 31 14:02:38.279253 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:02:38.279223    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1369bea_d9f3_4182_8a23_ee5a0a7c90a5.slice/crio-e45444d17cfd5a294ccb91d8b95f41f3486ee5a37596c0740ad6b1933d329e71 WatchSource:0}: Error finding container e45444d17cfd5a294ccb91d8b95f41f3486ee5a37596c0740ad6b1933d329e71: Status 404 returned error can't find the container with id e45444d17cfd5a294ccb91d8b95f41f3486ee5a37596c0740ad6b1933d329e71
Oct 31 14:02:38.484422 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:38.484394    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:38.682952 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:38.682906    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp" event={"ID":"b24885cc-c332-444e-89ba-9232a2b43196","Type":"ContainerStarted","Data":"ca3bc9dc2b8361179e7f548f47a82a0e00bddca315fc28f65e7737537cec79ee"}
Oct 31 14:02:38.683179 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:38.682966    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp" event={"ID":"b24885cc-c332-444e-89ba-9232a2b43196","Type":"ContainerStarted","Data":"287cf6702ff99231778f32de473feb417a49dc4db9e8195934cca54796941e09"}
Oct 31 14:02:38.685737 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:38.685708    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt" event={"ID":"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5","Type":"ContainerStarted","Data":"fe54ba873db8d09cb569a20ac6be0fa85a31102331fa8410dc3dcae229bcb9bd"}
Oct 31 14:02:38.685881 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:38.685755    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt" event={"ID":"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5","Type":"ContainerStarted","Data":"e45444d17cfd5a294ccb91d8b95f41f3486ee5a37596c0740ad6b1933d329e71"}
Oct 31 14:02:38.705915 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:38.705842    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp" podStartSLOduration=1.705828119 podStartE2EDuration="1.705828119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:38.70280332 +0000 UTC m=+3556.101151957" watchObservedRunningTime="2026-10-31 14:02:38.705828119 +0000 UTC m=+3556.104176743"
Oct 31 14:02:38.725075 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:38.725011    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt" podStartSLOduration=1.724990882 podStartE2EDuration="1.724990882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:38.722234178 +0000 UTC m=+3556.120582803" watchObservedRunningTime="2026-10-31 14:02:38.724990882 +0000 UTC m=+3556.123339507"
Oct 31 14:02:38.757925 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:38.757889    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b290805f-2d02-4d45-abbd-d0f0f8afe535" path="/var/lib/kubelet/pods/b290805f-2d02-4d45-abbd-d0f0f8afe535/volumes"
Oct 31 14:02:39.510846 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:39.510810    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:39.696782 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:39.696716    4428 generic.go:356] "Generic (PLEG): container finished" podID="a1369bea-d9f3-4182-8a23-ee5a0a7c90a5" containerID="fe54ba873db8d09cb569a20ac6be0fa85a31102331fa8410dc3dcae229bcb9bd" exitCode=0
Oct 31 14:02:39.697189 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:39.696821    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt" event={"ID":"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5","Type":"ContainerDied","Data":"fe54ba873db8d09cb569a20ac6be0fa85a31102331fa8410dc3dcae229bcb9bd"}
Oct 31 14:02:39.700168 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:39.700128    4428 generic.go:356] "Generic (PLEG): container finished" podID="b24885cc-c332-444e-89ba-9232a2b43196" containerID="ca3bc9dc2b8361179e7f548f47a82a0e00bddca315fc28f65e7737537cec79ee" exitCode=0
Oct 31 14:02:39.700347 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:39.700205    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp" event={"ID":"b24885cc-c332-444e-89ba-9232a2b43196","Type":"ContainerDied","Data":"ca3bc9dc2b8361179e7f548f47a82a0e00bddca315fc28f65e7737537cec79ee"}
Oct 31 14:02:40.027217 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.027170    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb" podUID="ee893209-7839-4a39-8aaa-17dc0f90d8d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:40.497234 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.497197    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:40.845909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.843001    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg" podUID="b042c863-4f10-4318-ac91-0464b31ebb89" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:40.888313 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.888277    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"
Oct 31 14:02:40.925826 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.925795    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"
Oct 31 14:02:40.989179 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.989149    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-kube-api-access-tdqgs\" (UniqueName: \"kubernetes.io/projected/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-kube-api-access-tdqgs\") pod \"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5\" (UID: \"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5\") "
Oct 31 14:02:40.989395 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.989237    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24885cc-c332-444e-89ba-9232a2b43196-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24885cc-c332-444e-89ba-9232a2b43196-maas-api-openshift-ca\") pod \"b24885cc-c332-444e-89ba-9232a2b43196\" (UID: \"b24885cc-c332-444e-89ba-9232a2b43196\") "
Oct 31 14:02:40.989395 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.989268    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b24885cc-c332-444e-89ba-9232a2b43196-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24885cc-c332-444e-89ba-9232a2b43196-maas-api-xks-ca\") pod \"b24885cc-c332-444e-89ba-9232a2b43196\" (UID: \"b24885cc-c332-444e-89ba-9232a2b43196\") "
Oct 31 14:02:40.989395 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.989299    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-maas-api-openshift-ca\") pod \"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5\" (UID: \"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5\") "
Oct 31 14:02:40.989395 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.989369    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-maas-api-xks-ca\") pod \"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5\" (UID: \"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5\") "
Oct 31 14:02:40.989647 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.989444    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24885cc-c332-444e-89ba-9232a2b43196-kube-api-access-x57qr\" (UniqueName: \"kubernetes.io/projected/b24885cc-c332-444e-89ba-9232a2b43196-kube-api-access-x57qr\") pod \"b24885cc-c332-444e-89ba-9232a2b43196\" (UID: \"b24885cc-c332-444e-89ba-9232a2b43196\") "
Oct 31 14:02:40.989715 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.989661    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24885cc-c332-444e-89ba-9232a2b43196-maas-api-openshift-ca" pod "b24885cc-c332-444e-89ba-9232a2b43196" (UID: "b24885cc-c332-444e-89ba-9232a2b43196"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:40.990074 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.990040    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-maas-api-openshift-ca" pod "a1369bea-d9f3-4182-8a23-ee5a0a7c90a5" (UID: "a1369bea-d9f3-4182-8a23-ee5a0a7c90a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:40.990203 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.990051    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24885cc-c332-444e-89ba-9232a2b43196-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:40.991796 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.991770    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b24885cc-c332-444e-89ba-9232a2b43196-maas-api-xks-ca" pod "b24885cc-c332-444e-89ba-9232a2b43196" (UID: "b24885cc-c332-444e-89ba-9232a2b43196"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:40.992012 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.991992    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24885cc-c332-444e-89ba-9232a2b43196-kube-api-access-x57qr" pod "b24885cc-c332-444e-89ba-9232a2b43196" (UID: "b24885cc-c332-444e-89ba-9232a2b43196"). InnerVolumeSpecName "kube-api-access-x57qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:40.992129 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.992111    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-kube-api-access-tdqgs" pod "a1369bea-d9f3-4182-8a23-ee5a0a7c90a5" (UID: "a1369bea-d9f3-4182-8a23-ee5a0a7c90a5"). InnerVolumeSpecName "kube-api-access-tdqgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:40.992462 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:40.992442    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-maas-api-xks-ca" pod "a1369bea-d9f3-4182-8a23-ee5a0a7c90a5" (UID: "a1369bea-d9f3-4182-8a23-ee5a0a7c90a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:41.034221 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.034120    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t" podUID="4ea944d0-8db8-4ae4-8e37-f04879e47a6a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:41.090917 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.090860    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdqgs\" (UniqueName: \"kubernetes.io/projected/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-kube-api-access-tdqgs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:41.090917 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.090911    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24885cc-c332-444e-89ba-9232a2b43196-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:41.090917 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.090922    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:41.091177 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.090930    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:41.091177 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.090939    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x57qr\" (UniqueName: \"kubernetes.io/projected/b24885cc-c332-444e-89ba-9232a2b43196-kube-api-access-x57qr\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:41.213998 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.213952    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main" containerID="cri-o://8481efb45d6fcdae9ac3557513c75466e9b7bb889bbd620f45d3d6206dadddab" gracePeriod=45
Oct 31 14:02:41.466942 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.466910    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 14:02:41.496338 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.496303    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-home\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-home\") pod \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") "
Oct 31 14:02:41.496534 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.496352    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1101d3a2-50ce-47fb-913c-4f40194fcdf1-tls-certs\" (UniqueName: \"kubernetes.io/secret/1101d3a2-50ce-47fb-913c-4f40194fcdf1-tls-certs\") pod \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") "
Oct 31 14:02:41.496534 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.496379    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-tmp-dir\") pod \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") "
Oct 31 14:02:41.496534 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.496421    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1101d3a2-50ce-47fb-913c-4f40194fcdf1-kube-api-access-2pvjg\" (UniqueName: \"kubernetes.io/projected/1101d3a2-50ce-47fb-913c-4f40194fcdf1-kube-api-access-2pvjg\") pod \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") "
Oct 31 14:02:41.496534 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.496449    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-model-cache\") pod \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") "
Oct 31 14:02:41.496767 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.496534    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-dshm\") pod \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\" (UID: \"1101d3a2-50ce-47fb-913c-4f40194fcdf1\") "
Oct 31 14:02:41.496767 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.496582    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-home" pod "1101d3a2-50ce-47fb-913c-4f40194fcdf1" (UID: "1101d3a2-50ce-47fb-913c-4f40194fcdf1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:41.496767 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.496720    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-tmp-dir" pod "1101d3a2-50ce-47fb-913c-4f40194fcdf1" (UID: "1101d3a2-50ce-47fb-913c-4f40194fcdf1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:41.498553 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.497359    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:41.498553 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.497390    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:41.498553 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.496779    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-model-cache" pod "1101d3a2-50ce-47fb-913c-4f40194fcdf1" (UID: "1101d3a2-50ce-47fb-913c-4f40194fcdf1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:41.499776 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.499752    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1101d3a2-50ce-47fb-913c-4f40194fcdf1-kube-api-access-2pvjg" pod "1101d3a2-50ce-47fb-913c-4f40194fcdf1" (UID: "1101d3a2-50ce-47fb-913c-4f40194fcdf1"). InnerVolumeSpecName "kube-api-access-2pvjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:41.500046 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.500014    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-dshm" pod "1101d3a2-50ce-47fb-913c-4f40194fcdf1" (UID: "1101d3a2-50ce-47fb-913c-4f40194fcdf1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:41.500624 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.500600    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1101d3a2-50ce-47fb-913c-4f40194fcdf1-tls-certs" pod "1101d3a2-50ce-47fb-913c-4f40194fcdf1" (UID: "1101d3a2-50ce-47fb-913c-4f40194fcdf1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:41.572277 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.572246    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:41.616806 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.605881    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1101d3a2-50ce-47fb-913c-4f40194fcdf1-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:41.616806 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.605949    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pvjg\" (UniqueName: \"kubernetes.io/projected/1101d3a2-50ce-47fb-913c-4f40194fcdf1-kube-api-access-2pvjg\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:41.616806 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.605964    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:41.616806 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.605973    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1101d3a2-50ce-47fb-913c-4f40194fcdf1-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:41.717166 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.717006    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"
Oct 31 14:02:41.717166 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.717037    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp" event={"ID":"b24885cc-c332-444e-89ba-9232a2b43196","Type":"ContainerDied","Data":"287cf6702ff99231778f32de473feb417a49dc4db9e8195934cca54796941e09"}
Oct 31 14:02:41.717166 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.717135    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="287cf6702ff99231778f32de473feb417a49dc4db9e8195934cca54796941e09"
Oct 31 14:02:41.721339 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.721303    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt" event={"ID":"a1369bea-d9f3-4182-8a23-ee5a0a7c90a5","Type":"ContainerDied","Data":"e45444d17cfd5a294ccb91d8b95f41f3486ee5a37596c0740ad6b1933d329e71"}
Oct 31 14:02:41.721522 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.721367    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e45444d17cfd5a294ccb91d8b95f41f3486ee5a37596c0740ad6b1933d329e71"
Oct 31 14:02:41.721522 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.721452    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"
Oct 31 14:02:41.727105 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.727070    4428 generic.go:356] "Generic (PLEG): container finished" podID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerID="8481efb45d6fcdae9ac3557513c75466e9b7bb889bbd620f45d3d6206dadddab" exitCode=0
Oct 31 14:02:41.727283 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.727123    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" event={"ID":"1101d3a2-50ce-47fb-913c-4f40194fcdf1","Type":"ContainerDied","Data":"8481efb45d6fcdae9ac3557513c75466e9b7bb889bbd620f45d3d6206dadddab"}
Oct 31 14:02:41.727283 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.727159    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv" event={"ID":"1101d3a2-50ce-47fb-913c-4f40194fcdf1","Type":"ContainerDied","Data":"eb1dd874a87dc5b54d87cc701106d819deeaa639eedc1ec23b71fae9790fd368"}
Oct 31 14:02:41.727283 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.727183    4428 scope.go:122] "RemoveContainer" containerID="8481efb45d6fcdae9ac3557513c75466e9b7bb889bbd620f45d3d6206dadddab"
Oct 31 14:02:41.727477 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.727322    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"
Oct 31 14:02:41.746828 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.746781    4428 scope.go:122] "RemoveContainer" containerID="8481efb45d6fcdae9ac3557513c75466e9b7bb889bbd620f45d3d6206dadddab"
Oct 31 14:02:41.747303 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:41.747243    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8481efb45d6fcdae9ac3557513c75466e9b7bb889bbd620f45d3d6206dadddab\": container with ID starting with 8481efb45d6fcdae9ac3557513c75466e9b7bb889bbd620f45d3d6206dadddab not found: ID does not exist" containerID="8481efb45d6fcdae9ac3557513c75466e9b7bb889bbd620f45d3d6206dadddab"
Oct 31 14:02:41.747428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.747290    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8481efb45d6fcdae9ac3557513c75466e9b7bb889bbd620f45d3d6206dadddab"} err="failed to get container status \"8481efb45d6fcdae9ac3557513c75466e9b7bb889bbd620f45d3d6206dadddab\": rpc error: code = NotFound desc = could not find container \"8481efb45d6fcdae9ac3557513c75466e9b7bb889bbd620f45d3d6206dadddab\": container with ID starting with 8481efb45d6fcdae9ac3557513c75466e9b7bb889bbd620f45d3d6206dadddab not found: ID does not exist"
Oct 31 14:02:41.761903 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.761849    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"]
Oct 31 14:02:41.767684 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.767656    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/premium-w3-c82140-kserve-6958b88b5-xxshv"]
Oct 31 14:02:41.855150 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:41.855114    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28" podUID="ba6a83e5-80ac-4dca-bbce-fefc479e0abe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:42.379650 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:42.379604    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"]
Oct 31 14:02:42.385753 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:42.385722    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"]
Oct 31 14:02:42.498576 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:42.498535    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:42.655616 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:42.654608    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:42.755851 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:42.755818    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" path="/var/lib/kubelet/pods/1101d3a2-50ce-47fb-913c-4f40194fcdf1/volumes"
Oct 31 14:02:42.816048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:42.816017    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:43.305402 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:43.305359    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:43.530467 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:43.530434    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"
Oct 31 14:02:44.104896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.103987    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:44.527073 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.527046    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"
Oct 31 14:02:44.654186 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.654087    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea944d0-8db8-4ae4-8e37-f04879e47a6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea944d0-8db8-4ae4-8e37-f04879e47a6a-plugins-config-volume\") pod \"4ea944d0-8db8-4ae4-8e37-f04879e47a6a\" (UID: \"4ea944d0-8db8-4ae4-8e37-f04879e47a6a\") "
Oct 31 14:02:44.654452 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.654287    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea944d0-8db8-4ae4-8e37-f04879e47a6a-kube-api-access-kpmh7\" (UniqueName: \"kubernetes.io/projected/4ea944d0-8db8-4ae4-8e37-f04879e47a6a-kube-api-access-kpmh7\") pod \"4ea944d0-8db8-4ae4-8e37-f04879e47a6a\" (UID: \"4ea944d0-8db8-4ae4-8e37-f04879e47a6a\") "
Oct 31 14:02:44.654563 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.654540    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea944d0-8db8-4ae4-8e37-f04879e47a6a-plugins-config-volume" pod "4ea944d0-8db8-4ae4-8e37-f04879e47a6a" (UID: "4ea944d0-8db8-4ae4-8e37-f04879e47a6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:44.654757 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.654740    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea944d0-8db8-4ae4-8e37-f04879e47a6a-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:44.656610 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.656590    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea944d0-8db8-4ae4-8e37-f04879e47a6a-kube-api-access-kpmh7" pod "4ea944d0-8db8-4ae4-8e37-f04879e47a6a" (UID: "4ea944d0-8db8-4ae4-8e37-f04879e47a6a"). InnerVolumeSpecName "kube-api-access-kpmh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:44.750285 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.750251    4428 generic.go:356] "Generic (PLEG): container finished" podID="4ea944d0-8db8-4ae4-8e37-f04879e47a6a" containerID="c0d48616b67d2a1af035c4edfebc61fba790923710cf6fefa99141887c42d7aa" exitCode=0
Oct 31 14:02:44.750475 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.750348    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"
Oct 31 14:02:44.755497 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.755461    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpmh7\" (UniqueName: \"kubernetes.io/projected/4ea944d0-8db8-4ae4-8e37-f04879e47a6a-kube-api-access-kpmh7\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:44.755728 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.755708    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t" event={"ID":"4ea944d0-8db8-4ae4-8e37-f04879e47a6a","Type":"ContainerDied","Data":"c0d48616b67d2a1af035c4edfebc61fba790923710cf6fefa99141887c42d7aa"}
Oct 31 14:02:44.755821 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.755752    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t" event={"ID":"4ea944d0-8db8-4ae4-8e37-f04879e47a6a","Type":"ContainerDied","Data":"66125c71ebc736d54d2af634dcf783fa9400c4d36b198ae1971d3b23682f7929"}
Oct 31 14:02:44.755821 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.755781    4428 scope.go:122] "RemoveContainer" containerID="c0d48616b67d2a1af035c4edfebc61fba790923710cf6fefa99141887c42d7aa"
Oct 31 14:02:44.766167 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.766144    4428 scope.go:122] "RemoveContainer" containerID="c0d48616b67d2a1af035c4edfebc61fba790923710cf6fefa99141887c42d7aa"
Oct 31 14:02:44.766493 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:44.766471    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d48616b67d2a1af035c4edfebc61fba790923710cf6fefa99141887c42d7aa\": container with ID starting with c0d48616b67d2a1af035c4edfebc61fba790923710cf6fefa99141887c42d7aa not found: ID does not exist" containerID="c0d48616b67d2a1af035c4edfebc61fba790923710cf6fefa99141887c42d7aa"
Oct 31 14:02:44.766557 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.766503    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d48616b67d2a1af035c4edfebc61fba790923710cf6fefa99141887c42d7aa"} err="failed to get container status \"c0d48616b67d2a1af035c4edfebc61fba790923710cf6fefa99141887c42d7aa\": rpc error: code = NotFound desc = could not find container \"c0d48616b67d2a1af035c4edfebc61fba790923710cf6fefa99141887c42d7aa\": container with ID starting with c0d48616b67d2a1af035c4edfebc61fba790923710cf6fefa99141887c42d7aa not found: ID does not exist"
Oct 31 14:02:44.781288 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.781255    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"]
Oct 31 14:02:44.792488 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.792451    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6784d6-d5845499-shd7t"]
Oct 31 14:02:44.931349 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:44.931296    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:45.117988 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.117959    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"
Oct 31 14:02:45.162541 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.160509    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b042c863-4f10-4318-ac91-0464b31ebb89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b042c863-4f10-4318-ac91-0464b31ebb89-plugins-config-volume\") pod \"b042c863-4f10-4318-ac91-0464b31ebb89\" (UID: \"b042c863-4f10-4318-ac91-0464b31ebb89\") "
Oct 31 14:02:45.162541 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.160610    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b042c863-4f10-4318-ac91-0464b31ebb89-kube-api-access-lc9ql\" (UniqueName: \"kubernetes.io/projected/b042c863-4f10-4318-ac91-0464b31ebb89-kube-api-access-lc9ql\") pod \"b042c863-4f10-4318-ac91-0464b31ebb89\" (UID: \"b042c863-4f10-4318-ac91-0464b31ebb89\") "
Oct 31 14:02:45.162541 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.161604    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b042c863-4f10-4318-ac91-0464b31ebb89-plugins-config-volume" pod "b042c863-4f10-4318-ac91-0464b31ebb89" (UID: "b042c863-4f10-4318-ac91-0464b31ebb89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:45.163745 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.163678    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b042c863-4f10-4318-ac91-0464b31ebb89-kube-api-access-lc9ql" pod "b042c863-4f10-4318-ac91-0464b31ebb89" (UID: "b042c863-4f10-4318-ac91-0464b31ebb89"). InnerVolumeSpecName "kube-api-access-lc9ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:45.174372 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.174341    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"
Oct 31 14:02:45.261774 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.261739    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6a83e5-80ac-4dca-bbce-fefc479e0abe-kube-api-access-bpmn6\" (UniqueName: \"kubernetes.io/projected/ba6a83e5-80ac-4dca-bbce-fefc479e0abe-kube-api-access-bpmn6\") pod \"ba6a83e5-80ac-4dca-bbce-fefc479e0abe\" (UID: \"ba6a83e5-80ac-4dca-bbce-fefc479e0abe\") "
Oct 31 14:02:45.262017 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.261936    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6a83e5-80ac-4dca-bbce-fefc479e0abe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6a83e5-80ac-4dca-bbce-fefc479e0abe-plugins-config-volume\") pod \"ba6a83e5-80ac-4dca-bbce-fefc479e0abe\" (UID: \"ba6a83e5-80ac-4dca-bbce-fefc479e0abe\") "
Oct 31 14:02:45.262302 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.262279    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6a83e5-80ac-4dca-bbce-fefc479e0abe-plugins-config-volume" pod "ba6a83e5-80ac-4dca-bbce-fefc479e0abe" (UID: "ba6a83e5-80ac-4dca-bbce-fefc479e0abe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:45.262425 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.262411    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b042c863-4f10-4318-ac91-0464b31ebb89-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:45.262471 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.262428    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6a83e5-80ac-4dca-bbce-fefc479e0abe-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:45.262471 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.262438    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc9ql\" (UniqueName: \"kubernetes.io/projected/b042c863-4f10-4318-ac91-0464b31ebb89-kube-api-access-lc9ql\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:45.264161 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.264138    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6a83e5-80ac-4dca-bbce-fefc479e0abe-kube-api-access-bpmn6" pod "ba6a83e5-80ac-4dca-bbce-fefc479e0abe" (UID: "ba6a83e5-80ac-4dca-bbce-fefc479e0abe"). InnerVolumeSpecName "kube-api-access-bpmn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:45.363145 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.363110    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpmn6\" (UniqueName: \"kubernetes.io/projected/ba6a83e5-80ac-4dca-bbce-fefc479e0abe-kube-api-access-bpmn6\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:45.758503 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.758464    4428 generic.go:356] "Generic (PLEG): container finished" podID="ba6a83e5-80ac-4dca-bbce-fefc479e0abe" containerID="3fb7763817dab9c9b09d1206e9a8b662fb121be571b2e77e3e22f579179097be" exitCode=0
Oct 31 14:02:45.758973 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.758516    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"
Oct 31 14:02:45.758973 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.758541    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28" event={"ID":"ba6a83e5-80ac-4dca-bbce-fefc479e0abe","Type":"ContainerDied","Data":"3fb7763817dab9c9b09d1206e9a8b662fb121be571b2e77e3e22f579179097be"}
Oct 31 14:02:45.758973 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.758601    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28" event={"ID":"ba6a83e5-80ac-4dca-bbce-fefc479e0abe","Type":"ContainerDied","Data":"d941dcc6455f2d360fcf295345c1854853f512cfb93c8cbf836a996bdbc6f036"}
Oct 31 14:02:45.758973 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.758626    4428 scope.go:122] "RemoveContainer" containerID="3fb7763817dab9c9b09d1206e9a8b662fb121be571b2e77e3e22f579179097be"
Oct 31 14:02:45.761460 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.761431    4428 generic.go:356] "Generic (PLEG): container finished" podID="b042c863-4f10-4318-ac91-0464b31ebb89" containerID="84ad66985cf7a6707d66b18f88efca9e49c2b28c522718d6a11eaebac3903cd4" exitCode=0
Oct 31 14:02:45.761574 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.761487    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"
Oct 31 14:02:45.761615 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.761485    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg" event={"ID":"b042c863-4f10-4318-ac91-0464b31ebb89","Type":"ContainerDied","Data":"84ad66985cf7a6707d66b18f88efca9e49c2b28c522718d6a11eaebac3903cd4"}
Oct 31 14:02:45.761668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.761612    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg" event={"ID":"b042c863-4f10-4318-ac91-0464b31ebb89","Type":"ContainerDied","Data":"efc8fa0be62cba4aca63be10693fd00ab9962414fdbb3ac1891390af9f398555"}
Oct 31 14:02:45.771550 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.771528    4428 scope.go:122] "RemoveContainer" containerID="3fb7763817dab9c9b09d1206e9a8b662fb121be571b2e77e3e22f579179097be"
Oct 31 14:02:45.771884 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:45.771837    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb7763817dab9c9b09d1206e9a8b662fb121be571b2e77e3e22f579179097be\": container with ID starting with 3fb7763817dab9c9b09d1206e9a8b662fb121be571b2e77e3e22f579179097be not found: ID does not exist" containerID="3fb7763817dab9c9b09d1206e9a8b662fb121be571b2e77e3e22f579179097be"
Oct 31 14:02:45.771983 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.771891    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb7763817dab9c9b09d1206e9a8b662fb121be571b2e77e3e22f579179097be"} err="failed to get container status \"3fb7763817dab9c9b09d1206e9a8b662fb121be571b2e77e3e22f579179097be\": rpc error: code = NotFound desc = could not find container \"3fb7763817dab9c9b09d1206e9a8b662fb121be571b2e77e3e22f579179097be\": container with ID starting with 3fb7763817dab9c9b09d1206e9a8b662fb121be571b2e77e3e22f579179097be not found: ID does not exist"
Oct 31 14:02:45.771983 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.771914    4428 scope.go:122] "RemoveContainer" containerID="84ad66985cf7a6707d66b18f88efca9e49c2b28c522718d6a11eaebac3903cd4"
Oct 31 14:02:45.781557 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.781531    4428 scope.go:122] "RemoveContainer" containerID="84ad66985cf7a6707d66b18f88efca9e49c2b28c522718d6a11eaebac3903cd4"
Oct 31 14:02:45.781847 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:45.781828    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ad66985cf7a6707d66b18f88efca9e49c2b28c522718d6a11eaebac3903cd4\": container with ID starting with 84ad66985cf7a6707d66b18f88efca9e49c2b28c522718d6a11eaebac3903cd4 not found: ID does not exist" containerID="84ad66985cf7a6707d66b18f88efca9e49c2b28c522718d6a11eaebac3903cd4"
Oct 31 14:02:45.781929 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.781855    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ad66985cf7a6707d66b18f88efca9e49c2b28c522718d6a11eaebac3903cd4"} err="failed to get container status \"84ad66985cf7a6707d66b18f88efca9e49c2b28c522718d6a11eaebac3903cd4\": rpc error: code = NotFound desc = could not find container \"84ad66985cf7a6707d66b18f88efca9e49c2b28c522718d6a11eaebac3903cd4\": container with ID starting with 84ad66985cf7a6707d66b18f88efca9e49c2b28c522718d6a11eaebac3903cd4 not found: ID does not exist"
Oct 31 14:02:45.788588 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.788554    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"]
Oct 31 14:02:45.794184 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.794157    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-5e8f29-5788d794dc-xtw28"]
Oct 31 14:02:45.807347 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.807315    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"]
Oct 31 14:02:45.810188 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.810168    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-5e8f29-94bb49845-c6tvg"]
Oct 31 14:02:45.946917 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:45.945582    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:46.111737 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.111634    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:46.406831 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.406736    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"]
Oct 31 14:02:46.407411 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407390    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b24885cc-c332-444e-89ba-9232a2b43196" containerName="revoke-keys"
Oct 31 14:02:46.407540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407413    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24885cc-c332-444e-89ba-9232a2b43196" containerName="revoke-keys"
Oct 31 14:02:46.407540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407427    4428 cpu_manager.go:414] "containerMap: removing container" podUID="ba6a83e5-80ac-4dca-bbce-fefc479e0abe" containerName="payload-pre-processing"
Oct 31 14:02:46.407540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407434    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6a83e5-80ac-4dca-bbce-fefc479e0abe" containerName="payload-pre-processing"
Oct 31 14:02:46.407540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407448    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4ea944d0-8db8-4ae4-8e37-f04879e47a6a" containerName="payload-pre-processing"
Oct 31 14:02:46.407540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407455    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea944d0-8db8-4ae4-8e37-f04879e47a6a" containerName="payload-pre-processing"
Oct 31 14:02:46.407540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407471    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b042c863-4f10-4318-ac91-0464b31ebb89" containerName="payload-processing"
Oct 31 14:02:46.407540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407479    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b042c863-4f10-4318-ac91-0464b31ebb89" containerName="payload-processing"
Oct 31 14:02:46.407540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407509    4428 cpu_manager.go:414] "containerMap: removing container" podUID="a1369bea-d9f3-4182-8a23-ee5a0a7c90a5" containerName="revoke-keys"
Oct 31 14:02:46.407540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407516    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1369bea-d9f3-4182-8a23-ee5a0a7c90a5" containerName="revoke-keys"
Oct 31 14:02:46.408072 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407548    4428 cpu_manager.go:414] "containerMap: removing container" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main"
Oct 31 14:02:46.408072 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407557    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main"
Oct 31 14:02:46.408072 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407696    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1369bea-d9f3-4182-8a23-ee5a0a7c90a5" containerName="revoke-keys"
Oct 31 14:02:46.408072 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407714    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1101d3a2-50ce-47fb-913c-4f40194fcdf1" containerName="main"
Oct 31 14:02:46.408072 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407727    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24885cc-c332-444e-89ba-9232a2b43196" containerName="revoke-keys"
Oct 31 14:02:46.408072 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407739    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6a83e5-80ac-4dca-bbce-fefc479e0abe" containerName="payload-pre-processing"
Oct 31 14:02:46.408072 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407749    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea944d0-8db8-4ae4-8e37-f04879e47a6a" containerName="payload-pre-processing"
Oct 31 14:02:46.408072 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.407759    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b042c863-4f10-4318-ac91-0464b31ebb89" containerName="payload-processing"
Oct 31 14:02:46.412738 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.412713    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"
Oct 31 14:02:46.419221 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.419195    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:02:46.419412 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.419296    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:02:46.423908 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.423881    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"]
Oct 31 14:02:46.477420 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.477383    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004b6efe-f6d2-4c88-bb66-d31445c02d73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-586469-cp9np\" (UID: \"004b6efe-f6d2-4c88-bb66-d31445c02d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"
Oct 31 14:02:46.477619 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.477439    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qcs\" (UniqueName: \"kubernetes.io/projected/004b6efe-f6d2-4c88-bb66-d31445c02d73-kube-api-access-m5qcs\") pod \"maas-api-revoke-keys-e2e-ipp-a-586469-cp9np\" (UID: \"004b6efe-f6d2-4c88-bb66-d31445c02d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"
Oct 31 14:02:46.477619 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.477577    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004b6efe-f6d2-4c88-bb66-d31445c02d73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-586469-cp9np\" (UID: \"004b6efe-f6d2-4c88-bb66-d31445c02d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"
Oct 31 14:02:46.578853 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.578794    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004b6efe-f6d2-4c88-bb66-d31445c02d73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-586469-cp9np\" (UID: \"004b6efe-f6d2-4c88-bb66-d31445c02d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"
Oct 31 14:02:46.578853 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.578884    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qcs\" (UniqueName: \"kubernetes.io/projected/004b6efe-f6d2-4c88-bb66-d31445c02d73-kube-api-access-m5qcs\") pod \"maas-api-revoke-keys-e2e-ipp-a-586469-cp9np\" (UID: \"004b6efe-f6d2-4c88-bb66-d31445c02d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"
Oct 31 14:02:46.579171 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.578931    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004b6efe-f6d2-4c88-bb66-d31445c02d73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-586469-cp9np\" (UID: \"004b6efe-f6d2-4c88-bb66-d31445c02d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"
Oct 31 14:02:46.579556 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.579526    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004b6efe-f6d2-4c88-bb66-d31445c02d73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-586469-cp9np\" (UID: \"004b6efe-f6d2-4c88-bb66-d31445c02d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"
Oct 31 14:02:46.581615 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.581589    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004b6efe-f6d2-4c88-bb66-d31445c02d73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-586469-cp9np\" (UID: \"004b6efe-f6d2-4c88-bb66-d31445c02d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"
Oct 31 14:02:46.593715 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.593687    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qcs\" (UniqueName: \"kubernetes.io/projected/004b6efe-f6d2-4c88-bb66-d31445c02d73-kube-api-access-m5qcs\") pod \"maas-api-revoke-keys-e2e-ipp-a-586469-cp9np\" (UID: \"004b6efe-f6d2-4c88-bb66-d31445c02d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"
Oct 31 14:02:46.729480 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.729449    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"
Oct 31 14:02:46.757546 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.757507    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea944d0-8db8-4ae4-8e37-f04879e47a6a" path="/var/lib/kubelet/pods/4ea944d0-8db8-4ae4-8e37-f04879e47a6a/volumes"
Oct 31 14:02:46.758002 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.757985    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b042c863-4f10-4318-ac91-0464b31ebb89" path="/var/lib/kubelet/pods/b042c863-4f10-4318-ac91-0464b31ebb89/volumes"
Oct 31 14:02:46.758387 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:46.758371    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6a83e5-80ac-4dca-bbce-fefc479e0abe" path="/var/lib/kubelet/pods/ba6a83e5-80ac-4dca-bbce-fefc479e0abe/volumes"
Oct 31 14:02:47.027180 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:47.027143    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"]
Oct 31 14:02:47.027618 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:02:47.027254    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004b6efe_f6d2_4c88_bb66_d31445c02d73.slice/crio-33c6a179f6621c1312952d604abb2582d476fbc4f986dab2616d706ee0366f65 WatchSource:0}: Error finding container 33c6a179f6621c1312952d604abb2582d476fbc4f986dab2616d706ee0366f65: Status 404 returned error can't find the container with id 33c6a179f6621c1312952d604abb2582d476fbc4f986dab2616d706ee0366f65
Oct 31 14:02:47.115924 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:47.115859    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:47.380997 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:47.380951    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp" podUID="7348227d-79bb-4bc8-b6ae-f4e318dcc703" containerName="payload-processing" containerID="cri-o://25bc270a2a63194bfe9a59e2adcef830df0af366088125fd7a6d2dbd36d38c27" gracePeriod=25
Oct 31 14:02:47.386676 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:47.386640    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w" podUID="b570ecdd-7dda-4021-966d-a36080ef5293" containerName="payload-pre-processing" containerID="cri-o://9aecfa60efd946312185571004b7166b43676f198ea380eab6cc580418e76dd7" gracePeriod=25
Oct 31 14:02:47.780003 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:47.779962    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np" event={"ID":"004b6efe-f6d2-4c88-bb66-d31445c02d73","Type":"ContainerStarted","Data":"041a1b26ddad0a92363f412581e5e242fe5b916c89d54f054e6da5e3480650e6"}
Oct 31 14:02:47.780219 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:47.780022    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np" event={"ID":"004b6efe-f6d2-4c88-bb66-d31445c02d73","Type":"ContainerStarted","Data":"33c6a179f6621c1312952d604abb2582d476fbc4f986dab2616d706ee0366f65"}
Oct 31 14:02:47.800325 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:47.800269    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np" podStartSLOduration=1.80025454 podStartE2EDuration="1.80025454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:47.798001674 +0000 UTC m=+3565.196350302" watchObservedRunningTime="2026-10-31 14:02:47.80025454 +0000 UTC m=+3565.198603165"
Oct 31 14:02:48.154352 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.154259    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"]
Oct 31 14:02:48.158300 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.158276    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"
Oct 31 14:02:48.170146 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.170115    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"]
Oct 31 14:02:48.198245 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.198210    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a6debd-03b9-4487-9d39-b9fa19078ea9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdb33258-s49k8\" (UID: \"07a6debd-03b9-4487-9d39-b9fa19078ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"
Oct 31 14:02:48.198428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.198261    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h2x6\" (UniqueName: \"kubernetes.io/projected/07a6debd-03b9-4487-9d39-b9fa19078ea9-kube-api-access-4h2x6\") pod \"maas-api-revoke-sub-fdb33258-s49k8\" (UID: \"07a6debd-03b9-4487-9d39-b9fa19078ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"
Oct 31 14:02:48.198479 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.198424    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a6debd-03b9-4487-9d39-b9fa19078ea9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdb33258-s49k8\" (UID: \"07a6debd-03b9-4487-9d39-b9fa19078ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"
Oct 31 14:02:48.299731 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.299687    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a6debd-03b9-4487-9d39-b9fa19078ea9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdb33258-s49k8\" (UID: \"07a6debd-03b9-4487-9d39-b9fa19078ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"
Oct 31 14:02:48.299731 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.299740    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h2x6\" (UniqueName: \"kubernetes.io/projected/07a6debd-03b9-4487-9d39-b9fa19078ea9-kube-api-access-4h2x6\") pod \"maas-api-revoke-sub-fdb33258-s49k8\" (UID: \"07a6debd-03b9-4487-9d39-b9fa19078ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"
Oct 31 14:02:48.299987 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.299787    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a6debd-03b9-4487-9d39-b9fa19078ea9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdb33258-s49k8\" (UID: \"07a6debd-03b9-4487-9d39-b9fa19078ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"
Oct 31 14:02:48.300461 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.300423    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a6debd-03b9-4487-9d39-b9fa19078ea9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdb33258-s49k8\" (UID: \"07a6debd-03b9-4487-9d39-b9fa19078ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"
Oct 31 14:02:48.302415 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.302393    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a6debd-03b9-4487-9d39-b9fa19078ea9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdb33258-s49k8\" (UID: \"07a6debd-03b9-4487-9d39-b9fa19078ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"
Oct 31 14:02:48.316615 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.316576    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h2x6\" (UniqueName: \"kubernetes.io/projected/07a6debd-03b9-4487-9d39-b9fa19078ea9-kube-api-access-4h2x6\") pod \"maas-api-revoke-sub-fdb33258-s49k8\" (UID: \"07a6debd-03b9-4487-9d39-b9fa19078ea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"
Oct 31 14:02:48.475003 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.474963    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"
Oct 31 14:02:48.565537 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.565461    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:48.741999 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:48.741914    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:02:48.771845 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.771818    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"]
Oct 31 14:02:48.774995 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:02:48.774966    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a6debd_03b9_4487_9d39_b9fa19078ea9.slice/crio-40ca2c457b72bdf33c0d02cfa8baf7f5af35fa7b96645c0baf332e17dac3a0b2 WatchSource:0}: Error finding container 40ca2c457b72bdf33c0d02cfa8baf7f5af35fa7b96645c0baf332e17dac3a0b2: Status 404 returned error can't find the container with id 40ca2c457b72bdf33c0d02cfa8baf7f5af35fa7b96645c0baf332e17dac3a0b2
Oct 31 14:02:48.787113 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.787081    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8" event={"ID":"07a6debd-03b9-4487-9d39-b9fa19078ea9","Type":"ContainerStarted","Data":"40ca2c457b72bdf33c0d02cfa8baf7f5af35fa7b96645c0baf332e17dac3a0b2"}
Oct 31 14:02:48.788713 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.788688    4428 generic.go:356] "Generic (PLEG): container finished" podID="004b6efe-f6d2-4c88-bb66-d31445c02d73" containerID="041a1b26ddad0a92363f412581e5e242fe5b916c89d54f054e6da5e3480650e6" exitCode=0
Oct 31 14:02:48.788837 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:48.788755    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np" event={"ID":"004b6efe-f6d2-4c88-bb66-d31445c02d73","Type":"ContainerDied","Data":"041a1b26ddad0a92363f412581e5e242fe5b916c89d54f054e6da5e3480650e6"}
Oct 31 14:02:49.544585 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:49.544512    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:49.796837 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:49.796746    4428 generic.go:356] "Generic (PLEG): container finished" podID="07a6debd-03b9-4487-9d39-b9fa19078ea9" containerID="a24cd72626669a4cf831e93aaa0eb477e2270bbd4d94b1ca209e520fa6db1e42" exitCode=0
Oct 31 14:02:49.797025 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:49.796832    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8" event={"ID":"07a6debd-03b9-4487-9d39-b9fa19078ea9","Type":"ContainerDied","Data":"a24cd72626669a4cf831e93aaa0eb477e2270bbd4d94b1ca209e520fa6db1e42"}
Oct 31 14:02:49.946984 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:49.946954    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"
Oct 31 14:02:50.020747 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:50.020708    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/004b6efe-f6d2-4c88-bb66-d31445c02d73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004b6efe-f6d2-4c88-bb66-d31445c02d73-maas-api-xks-ca\") pod \"004b6efe-f6d2-4c88-bb66-d31445c02d73\" (UID: \"004b6efe-f6d2-4c88-bb66-d31445c02d73\") "
Oct 31 14:02:50.021008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:50.020775    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/004b6efe-f6d2-4c88-bb66-d31445c02d73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004b6efe-f6d2-4c88-bb66-d31445c02d73-maas-api-openshift-ca\") pod \"004b6efe-f6d2-4c88-bb66-d31445c02d73\" (UID: \"004b6efe-f6d2-4c88-bb66-d31445c02d73\") "
Oct 31 14:02:50.021008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:50.020964    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004b6efe-f6d2-4c88-bb66-d31445c02d73-kube-api-access-m5qcs\" (UniqueName: \"kubernetes.io/projected/004b6efe-f6d2-4c88-bb66-d31445c02d73-kube-api-access-m5qcs\") pod \"004b6efe-f6d2-4c88-bb66-d31445c02d73\" (UID: \"004b6efe-f6d2-4c88-bb66-d31445c02d73\") "
Oct 31 14:02:50.021580 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:50.021315    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004b6efe-f6d2-4c88-bb66-d31445c02d73-maas-api-openshift-ca" pod "004b6efe-f6d2-4c88-bb66-d31445c02d73" (UID: "004b6efe-f6d2-4c88-bb66-d31445c02d73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:50.022554 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:50.022524    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004b6efe-f6d2-4c88-bb66-d31445c02d73-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:50.024378 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:50.024349    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004b6efe-f6d2-4c88-bb66-d31445c02d73-kube-api-access-m5qcs" pod "004b6efe-f6d2-4c88-bb66-d31445c02d73" (UID: "004b6efe-f6d2-4c88-bb66-d31445c02d73"). InnerVolumeSpecName "kube-api-access-m5qcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:50.024486 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:50.024379    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/004b6efe-f6d2-4c88-bb66-d31445c02d73-maas-api-xks-ca" pod "004b6efe-f6d2-4c88-bb66-d31445c02d73" (UID: "004b6efe-f6d2-4c88-bb66-d31445c02d73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:50.027628 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:50.027604    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb" podUID="ee893209-7839-4a39-8aaa-17dc0f90d8d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:50.123854 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:50.123765    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004b6efe-f6d2-4c88-bb66-d31445c02d73-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:50.123854 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:50.123797    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5qcs\" (UniqueName: \"kubernetes.io/projected/004b6efe-f6d2-4c88-bb66-d31445c02d73-kube-api-access-m5qcs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:50.573705 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:50.573675    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:50.807091 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:50.807061    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"
Oct 31 14:02:50.807266 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:50.807088    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np" event={"ID":"004b6efe-f6d2-4c88-bb66-d31445c02d73","Type":"ContainerDied","Data":"33c6a179f6621c1312952d604abb2582d476fbc4f986dab2616d706ee0366f65"}
Oct 31 14:02:50.807266 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:50.807151    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33c6a179f6621c1312952d604abb2582d476fbc4f986dab2616d706ee0366f65"
Oct 31 14:02:50.984228 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:50.984204    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"
Oct 31 14:02:51.035370 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.035329    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a6debd-03b9-4487-9d39-b9fa19078ea9-kube-api-access-4h2x6\" (UniqueName: \"kubernetes.io/projected/07a6debd-03b9-4487-9d39-b9fa19078ea9-kube-api-access-4h2x6\") pod \"07a6debd-03b9-4487-9d39-b9fa19078ea9\" (UID: \"07a6debd-03b9-4487-9d39-b9fa19078ea9\") "
Oct 31 14:02:51.035370 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.035386    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a6debd-03b9-4487-9d39-b9fa19078ea9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a6debd-03b9-4487-9d39-b9fa19078ea9-maas-api-openshift-ca\") pod \"07a6debd-03b9-4487-9d39-b9fa19078ea9\" (UID: \"07a6debd-03b9-4487-9d39-b9fa19078ea9\") "
Oct 31 14:02:51.035653 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.035455    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a6debd-03b9-4487-9d39-b9fa19078ea9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a6debd-03b9-4487-9d39-b9fa19078ea9-maas-api-xks-ca\") pod \"07a6debd-03b9-4487-9d39-b9fa19078ea9\" (UID: \"07a6debd-03b9-4487-9d39-b9fa19078ea9\") "
Oct 31 14:02:51.040386 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.040346    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:51.046091 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.042830    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a6debd-03b9-4487-9d39-b9fa19078ea9-maas-api-openshift-ca" pod "07a6debd-03b9-4487-9d39-b9fa19078ea9" (UID: "07a6debd-03b9-4487-9d39-b9fa19078ea9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:51.046605 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.046570    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a6debd-03b9-4487-9d39-b9fa19078ea9-kube-api-access-4h2x6" pod "07a6debd-03b9-4487-9d39-b9fa19078ea9" (UID: "07a6debd-03b9-4487-9d39-b9fa19078ea9"). InnerVolumeSpecName "kube-api-access-4h2x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:51.068427 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.061641    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a6debd-03b9-4487-9d39-b9fa19078ea9-maas-api-xks-ca" pod "07a6debd-03b9-4487-9d39-b9fa19078ea9" (UID: "07a6debd-03b9-4487-9d39-b9fa19078ea9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:51.137969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.137845    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h2x6\" (UniqueName: \"kubernetes.io/projected/07a6debd-03b9-4487-9d39-b9fa19078ea9-kube-api-access-4h2x6\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:51.137969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.137893    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a6debd-03b9-4487-9d39-b9fa19078ea9-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:51.137969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.137906    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a6debd-03b9-4487-9d39-b9fa19078ea9-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:51.220069 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.220010    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:51.393246 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.393160    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:51.550801 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.550758    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:51.694168 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.694071    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"]
Oct 31 14:02:51.701756 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.701721    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c82140-k6shg"]
Oct 31 14:02:51.726780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.726735    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:51.815215 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.815180    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"
Oct 31 14:02:51.815215 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.815207    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8" event={"ID":"07a6debd-03b9-4487-9d39-b9fa19078ea9","Type":"ContainerDied","Data":"40ca2c457b72bdf33c0d02cfa8baf7f5af35fa7b96645c0baf332e17dac3a0b2"}
Oct 31 14:02:51.815468 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.815254    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40ca2c457b72bdf33c0d02cfa8baf7f5af35fa7b96645c0baf332e17dac3a0b2"
Oct 31 14:02:51.896530 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:51.896496    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:52.172342 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.172277    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main" containerID="cri-o://2cf5db9e407957590da9856d40402d7f20a7c9b83d442497fcee1d7912473abb" gracePeriod=45
Oct 31 14:02:52.187453 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.187413    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main" containerID="cri-o://deab884524b3c7337469c5258a4440cf3037e44583dddc6ac06da2185285c568" gracePeriod=45
Oct 31 14:02:52.246430 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.246391    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:52.476368 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.476333    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 14:02:52.510909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.510848    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 14:02:52.568480 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.568431    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d6a586d-e1a2-4965-b0fc-89e2286d7298-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6a586d-e1a2-4965-b0fc-89e2286d7298-tls-certs\") pod \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") "
Oct 31 14:02:52.568693 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.568558    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-home\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-home\") pod \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") "
Oct 31 14:02:52.568693 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.568608    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-home\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-home\") pod \"b950b997-7653-480d-b120-31dc384832d0\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") "
Oct 31 14:02:52.568693 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.568641    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-tmp-dir\") pod \"b950b997-7653-480d-b120-31dc384832d0\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") "
Oct 31 14:02:52.568693 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.568679    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b950b997-7653-480d-b120-31dc384832d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/b950b997-7653-480d-b120-31dc384832d0-tls-certs\") pod \"b950b997-7653-480d-b120-31dc384832d0\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") "
Oct 31 14:02:52.568985 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.568708    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-dshm\") pod \"b950b997-7653-480d-b120-31dc384832d0\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") "
Oct 31 14:02:52.568985 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.568789    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-model-cache\") pod \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") "
Oct 31 14:02:52.568985 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.568833    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6a586d-e1a2-4965-b0fc-89e2286d7298-kube-api-access-llrqj\" (UniqueName: \"kubernetes.io/projected/5d6a586d-e1a2-4965-b0fc-89e2286d7298-kube-api-access-llrqj\") pod \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") "
Oct 31 14:02:52.568985 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.568887    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-home" pod "5d6a586d-e1a2-4965-b0fc-89e2286d7298" (UID: "5d6a586d-e1a2-4965-b0fc-89e2286d7298"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:52.568985 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.568901    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-home" pod "b950b997-7653-480d-b120-31dc384832d0" (UID: "b950b997-7653-480d-b120-31dc384832d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:52.568985 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.568925    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b950b997-7653-480d-b120-31dc384832d0-kube-api-access-rcbc9\" (UniqueName: \"kubernetes.io/projected/b950b997-7653-480d-b120-31dc384832d0-kube-api-access-rcbc9\") pod \"b950b997-7653-480d-b120-31dc384832d0\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") "
Oct 31 14:02:52.569328 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.569115    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-model-cache" pod "5d6a586d-e1a2-4965-b0fc-89e2286d7298" (UID: "5d6a586d-e1a2-4965-b0fc-89e2286d7298"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:52.569328 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.569159    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-dshm\") pod \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") "
Oct 31 14:02:52.569328 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.569235    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-model-cache\") pod \"b950b997-7653-480d-b120-31dc384832d0\" (UID: \"b950b997-7653-480d-b120-31dc384832d0\") "
Oct 31 14:02:52.569328 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.569297    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-tmp-dir\") pod \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\" (UID: \"5d6a586d-e1a2-4965-b0fc-89e2286d7298\") "
Oct 31 14:02:52.570063 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.570028    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-tmp-dir" pod "b950b997-7653-480d-b120-31dc384832d0" (UID: "b950b997-7653-480d-b120-31dc384832d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:52.570178 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.570086    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.570178 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.570106    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.570178 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.570125    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.570364 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.570262    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-tmp-dir" pod "5d6a586d-e1a2-4965-b0fc-89e2286d7298" (UID: "5d6a586d-e1a2-4965-b0fc-89e2286d7298"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:52.570428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.570404    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-model-cache" pod "b950b997-7653-480d-b120-31dc384832d0" (UID: "b950b997-7653-480d-b120-31dc384832d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:52.572222 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.572191    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b950b997-7653-480d-b120-31dc384832d0-kube-api-access-rcbc9" pod "b950b997-7653-480d-b120-31dc384832d0" (UID: "b950b997-7653-480d-b120-31dc384832d0"). InnerVolumeSpecName "kube-api-access-rcbc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:52.572527 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.572500    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-dshm" pod "b950b997-7653-480d-b120-31dc384832d0" (UID: "b950b997-7653-480d-b120-31dc384832d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:52.573675 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.573640    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6a586d-e1a2-4965-b0fc-89e2286d7298-kube-api-access-llrqj" pod "5d6a586d-e1a2-4965-b0fc-89e2286d7298" (UID: "5d6a586d-e1a2-4965-b0fc-89e2286d7298"). InnerVolumeSpecName "kube-api-access-llrqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:52.573920 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.573892    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b950b997-7653-480d-b120-31dc384832d0-tls-certs" pod "b950b997-7653-480d-b120-31dc384832d0" (UID: "b950b997-7653-480d-b120-31dc384832d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:52.574070 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.573934    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d6a586d-e1a2-4965-b0fc-89e2286d7298-tls-certs" pod "5d6a586d-e1a2-4965-b0fc-89e2286d7298" (UID: "5d6a586d-e1a2-4965-b0fc-89e2286d7298"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:52.574349 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.574321    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-dshm" pod "5d6a586d-e1a2-4965-b0fc-89e2286d7298" (UID: "5d6a586d-e1a2-4965-b0fc-89e2286d7298"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:02:52.671009 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.670974    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.671009 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.671006    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6a586d-e1a2-4965-b0fc-89e2286d7298-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.671009 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.671015    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.671288 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.671024    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b950b997-7653-480d-b120-31dc384832d0-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.671288 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.671033    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.671288 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.671042    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llrqj\" (UniqueName: \"kubernetes.io/projected/5d6a586d-e1a2-4965-b0fc-89e2286d7298-kube-api-access-llrqj\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.671288 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.671052    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcbc9\" (UniqueName: \"kubernetes.io/projected/b950b997-7653-480d-b120-31dc384832d0-kube-api-access-rcbc9\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.671288 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.671062    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6a586d-e1a2-4965-b0fc-89e2286d7298-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.671288 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.671071    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b950b997-7653-480d-b120-31dc384832d0-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:52.755946 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.755911    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d664724b-e6a5-4cad-8317-b97fcfa2e0fe" path="/var/lib/kubelet/pods/d664724b-e6a5-4cad-8317-b97fcfa2e0fe/volumes"
Oct 31 14:02:52.823223 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.823184    4428 generic.go:356] "Generic (PLEG): container finished" podID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerID="deab884524b3c7337469c5258a4440cf3037e44583dddc6ac06da2185285c568" exitCode=0
Oct 31 14:02:52.823407 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.823243    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"
Oct 31 14:02:52.823407 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.823267    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" event={"ID":"5d6a586d-e1a2-4965-b0fc-89e2286d7298","Type":"ContainerDied","Data":"deab884524b3c7337469c5258a4440cf3037e44583dddc6ac06da2185285c568"}
Oct 31 14:02:52.823407 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.823315    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn" event={"ID":"5d6a586d-e1a2-4965-b0fc-89e2286d7298","Type":"ContainerDied","Data":"7f9446ccf3fad8d35639e7b591e24955d0e85c803f332e069231181050f7bd25"}
Oct 31 14:02:52.823407 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.823334    4428 scope.go:122] "RemoveContainer" containerID="deab884524b3c7337469c5258a4440cf3037e44583dddc6ac06da2185285c568"
Oct 31 14:02:52.825553 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.825402    4428 generic.go:356] "Generic (PLEG): container finished" podID="b950b997-7653-480d-b120-31dc384832d0" containerID="2cf5db9e407957590da9856d40402d7f20a7c9b83d442497fcee1d7912473abb" exitCode=0
Oct 31 14:02:52.825823 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.825477    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"
Oct 31 14:02:52.825955 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.825495    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" event={"ID":"b950b997-7653-480d-b120-31dc384832d0","Type":"ContainerDied","Data":"2cf5db9e407957590da9856d40402d7f20a7c9b83d442497fcee1d7912473abb"}
Oct 31 14:02:52.825955 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.825926    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm" event={"ID":"b950b997-7653-480d-b120-31dc384832d0","Type":"ContainerDied","Data":"7e4a3d8e1e53fbd4c81ff4a6810e84f22c2722cd9afdd3c8c53790e012ad8d84"}
Oct 31 14:02:52.834377 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.834354    4428 scope.go:122] "RemoveContainer" containerID="deab884524b3c7337469c5258a4440cf3037e44583dddc6ac06da2185285c568"
Oct 31 14:02:52.834682 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:52.834663    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deab884524b3c7337469c5258a4440cf3037e44583dddc6ac06da2185285c568\": container with ID starting with deab884524b3c7337469c5258a4440cf3037e44583dddc6ac06da2185285c568 not found: ID does not exist" containerID="deab884524b3c7337469c5258a4440cf3037e44583dddc6ac06da2185285c568"
Oct 31 14:02:52.834757 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.834696    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deab884524b3c7337469c5258a4440cf3037e44583dddc6ac06da2185285c568"} err="failed to get container status \"deab884524b3c7337469c5258a4440cf3037e44583dddc6ac06da2185285c568\": rpc error: code = NotFound desc = could not find container \"deab884524b3c7337469c5258a4440cf3037e44583dddc6ac06da2185285c568\": container with ID starting with deab884524b3c7337469c5258a4440cf3037e44583dddc6ac06da2185285c568 not found: ID does not exist"
Oct 31 14:02:52.834757 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.834714    4428 scope.go:122] "RemoveContainer" containerID="2cf5db9e407957590da9856d40402d7f20a7c9b83d442497fcee1d7912473abb"
Oct 31 14:02:52.844497 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.844473    4428 scope.go:122] "RemoveContainer" containerID="2cf5db9e407957590da9856d40402d7f20a7c9b83d442497fcee1d7912473abb"
Oct 31 14:02:52.844843 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:52.844823    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf5db9e407957590da9856d40402d7f20a7c9b83d442497fcee1d7912473abb\": container with ID starting with 2cf5db9e407957590da9856d40402d7f20a7c9b83d442497fcee1d7912473abb not found: ID does not exist" containerID="2cf5db9e407957590da9856d40402d7f20a7c9b83d442497fcee1d7912473abb"
Oct 31 14:02:52.844935 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.844851    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf5db9e407957590da9856d40402d7f20a7c9b83d442497fcee1d7912473abb"} err="failed to get container status \"2cf5db9e407957590da9856d40402d7f20a7c9b83d442497fcee1d7912473abb\": rpc error: code = NotFound desc = could not find container \"2cf5db9e407957590da9856d40402d7f20a7c9b83d442497fcee1d7912473abb\": container with ID starting with 2cf5db9e407957590da9856d40402d7f20a7c9b83d442497fcee1d7912473abb not found: ID does not exist"
Oct 31 14:02:52.849806 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.849754    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"]
Oct 31 14:02:52.853153 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.853127    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/distinct-2-w6-aabeef-kserve-b7ddcd476-qn8sm"]
Oct 31 14:02:52.866679 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.866650    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"]
Oct 31 14:02:52.870492 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:52.870466    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/premium-w6-aabeef-kserve-575d86888d-7hvjn"]
Oct 31 14:02:53.053122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:53.053025    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp" podUID="7348227d-79bb-4bc8-b6ae-f4e318dcc703" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:53.053122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:53.053054    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w" podUID="b570ecdd-7dda-4021-966d-a36080ef5293" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:02:53.342387 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:53.342302    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:54.308375 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:54.308328    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:54.474591 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:54.474541    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:54.754999 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:54.754969    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" path="/var/lib/kubelet/pods/5d6a586d-e1a2-4965-b0fc-89e2286d7298/volumes"
Oct 31 14:02:54.755428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:54.755414    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b950b997-7653-480d-b120-31dc384832d0" path="/var/lib/kubelet/pods/b950b997-7653-480d-b120-31dc384832d0/volumes"
Oct 31 14:02:55.252663 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:55.252627    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:56.036938 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.036897    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:56.715921 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.715897    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"
Oct 31 14:02:56.822845 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.822814    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee893209-7839-4a39-8aaa-17dc0f90d8d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee893209-7839-4a39-8aaa-17dc0f90d8d7-plugins-config-volume\") pod \"ee893209-7839-4a39-8aaa-17dc0f90d8d7\" (UID: \"ee893209-7839-4a39-8aaa-17dc0f90d8d7\") "
Oct 31 14:02:56.823074 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.823048    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee893209-7839-4a39-8aaa-17dc0f90d8d7-kube-api-access-cwj95\" (UniqueName: \"kubernetes.io/projected/ee893209-7839-4a39-8aaa-17dc0f90d8d7-kube-api-access-cwj95\") pod \"ee893209-7839-4a39-8aaa-17dc0f90d8d7\" (UID: \"ee893209-7839-4a39-8aaa-17dc0f90d8d7\") "
Oct 31 14:02:56.823213 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.823192    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee893209-7839-4a39-8aaa-17dc0f90d8d7-plugins-config-volume" pod "ee893209-7839-4a39-8aaa-17dc0f90d8d7" (UID: "ee893209-7839-4a39-8aaa-17dc0f90d8d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:56.825531 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.825489    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee893209-7839-4a39-8aaa-17dc0f90d8d7-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:56.826602 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.826546    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee893209-7839-4a39-8aaa-17dc0f90d8d7-kube-api-access-cwj95" pod "ee893209-7839-4a39-8aaa-17dc0f90d8d7" (UID: "ee893209-7839-4a39-8aaa-17dc0f90d8d7"). InnerVolumeSpecName "kube-api-access-cwj95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:56.855835 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.855799    4428 generic.go:356] "Generic (PLEG): container finished" podID="ee893209-7839-4a39-8aaa-17dc0f90d8d7" containerID="bc0ed2137f062dbff31cbab5f0d1f1804312409d8a3ed243ae582fa39c7b2e44" exitCode=0
Oct 31 14:02:56.856001 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.855855    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb" event={"ID":"ee893209-7839-4a39-8aaa-17dc0f90d8d7","Type":"ContainerDied","Data":"bc0ed2137f062dbff31cbab5f0d1f1804312409d8a3ed243ae582fa39c7b2e44"}
Oct 31 14:02:56.856001 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.855860    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"
Oct 31 14:02:56.856001 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.855947    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb" event={"ID":"ee893209-7839-4a39-8aaa-17dc0f90d8d7","Type":"ContainerDied","Data":"69c5a29bb00ed250e35a8e5272a6e0b8e270197674189c4e836d5e76780c0fe1"}
Oct 31 14:02:56.856001 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.855971    4428 scope.go:122] "RemoveContainer" containerID="bc0ed2137f062dbff31cbab5f0d1f1804312409d8a3ed243ae582fa39c7b2e44"
Oct 31 14:02:56.867178 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.867152    4428 scope.go:122] "RemoveContainer" containerID="bc0ed2137f062dbff31cbab5f0d1f1804312409d8a3ed243ae582fa39c7b2e44"
Oct 31 14:02:56.867488 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:56.867467    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0ed2137f062dbff31cbab5f0d1f1804312409d8a3ed243ae582fa39c7b2e44\": container with ID starting with bc0ed2137f062dbff31cbab5f0d1f1804312409d8a3ed243ae582fa39c7b2e44 not found: ID does not exist" containerID="bc0ed2137f062dbff31cbab5f0d1f1804312409d8a3ed243ae582fa39c7b2e44"
Oct 31 14:02:56.867562 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.867501    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0ed2137f062dbff31cbab5f0d1f1804312409d8a3ed243ae582fa39c7b2e44"} err="failed to get container status \"bc0ed2137f062dbff31cbab5f0d1f1804312409d8a3ed243ae582fa39c7b2e44\": rpc error: code = NotFound desc = could not find container \"bc0ed2137f062dbff31cbab5f0d1f1804312409d8a3ed243ae582fa39c7b2e44\": container with ID starting with bc0ed2137f062dbff31cbab5f0d1f1804312409d8a3ed243ae582fa39c7b2e44 not found: ID does not exist"
Oct 31 14:02:56.887568 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.883962    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"]
Oct 31 14:02:56.890843 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.890814    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c82140-68b5fb4d55-8w8nb"]
Oct 31 14:02:56.926685 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.926650    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwj95\" (UniqueName: \"kubernetes.io/projected/ee893209-7839-4a39-8aaa-17dc0f90d8d7-kube-api-access-cwj95\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:56.986942 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:56.986511    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:57.855529 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:57.855492    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:58.754409 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:58.754380    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee893209-7839-4a39-8aaa-17dc0f90d8d7" path="/var/lib/kubelet/pods/ee893209-7839-4a39-8aaa-17dc0f90d8d7/volumes"
Oct 31 14:02:58.802849 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:58.802793    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:59.329133 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.329097    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"]
Oct 31 14:02:59.329689 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.329329    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl" podUID="ed8f2762-7f5a-4bf8-b40f-981f560b4e28" containerName="maas-api" containerID="cri-o://d6d64fd454f4dabfbaf1894863dba1a684b33b0af296a905ef07b9ec02fcd058" gracePeriod=30
Oct 31 14:02:59.418294 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.418255    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"]
Oct 31 14:02:59.418514 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.418487    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt" podUID="4e024838-48ac-410b-8cce-cfb5f6ee6ef9" containerName="payload-processing" containerID="cri-o://b1e840fee4bd4d02a1ecf5f42995f96d34f8a3dee4ad3c3219f4c5a1991198cd" gracePeriod=30
Oct 31 14:02:59.431597 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.431560    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"]
Oct 31 14:02:59.431912 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.431881    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj" podUID="1c37feae-b089-45dd-a74e-d52516c48a67" containerName="payload-pre-processing" containerID="cri-o://392317b6cb6029d6023b2ecfb6eebe05b0a075fb5f850a352c7866fd33d459f2" gracePeriod=30
Oct 31 14:02:59.450557 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.450504    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:59.610909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.609393    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:59.776851 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.776821    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:02:59.779766 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.779712    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:02:59.789693 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.789657    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"]
Oct 31 14:02:59.793492 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.793405    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-586469-cp9np"]
Oct 31 14:02:59.800794 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.800733    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"
Oct 31 14:02:59.846699 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.846617    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"
Oct 31 14:02:59.873685 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.873649    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-metrics-certs\") pod \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\" (UID: \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\") "
Oct 31 14:02:59.874084 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.874053    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-kube-api-access-7rztr\" (UniqueName: \"kubernetes.io/projected/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-kube-api-access-7rztr\") pod \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\" (UID: \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\") "
Oct 31 14:02:59.874365 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.874335    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-maas-api-tls\") pod \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\" (UID: \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\") "
Oct 31 14:02:59.874581 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.874566    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-openshift-service-ca\") pod \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\" (UID: \"ed8f2762-7f5a-4bf8-b40f-981f560b4e28\") "
Oct 31 14:02:59.875687 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.875649    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-openshift-service-ca" pod "ed8f2762-7f5a-4bf8-b40f-981f560b4e28" (UID: "ed8f2762-7f5a-4bf8-b40f-981f560b4e28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:59.887105 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.887066    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-maas-api-tls" pod "ed8f2762-7f5a-4bf8-b40f-981f560b4e28" (UID: "ed8f2762-7f5a-4bf8-b40f-981f560b4e28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:59.889090 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.889058    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-metrics-certs" pod "ed8f2762-7f5a-4bf8-b40f-981f560b4e28" (UID: "ed8f2762-7f5a-4bf8-b40f-981f560b4e28"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:02:59.889651 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.889605    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-kube-api-access-7rztr" pod "ed8f2762-7f5a-4bf8-b40f-981f560b4e28" (UID: "ed8f2762-7f5a-4bf8-b40f-981f560b4e28"). InnerVolumeSpecName "kube-api-access-7rztr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:59.890276 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.890132    4428 generic.go:356] "Generic (PLEG): container finished" podID="1c37feae-b089-45dd-a74e-d52516c48a67" containerID="392317b6cb6029d6023b2ecfb6eebe05b0a075fb5f850a352c7866fd33d459f2" exitCode=0
Oct 31 14:02:59.890276 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.890206    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj" event={"ID":"1c37feae-b089-45dd-a74e-d52516c48a67","Type":"ContainerDied","Data":"392317b6cb6029d6023b2ecfb6eebe05b0a075fb5f850a352c7866fd33d459f2"}
Oct 31 14:02:59.890463 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.890280    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj" event={"ID":"1c37feae-b089-45dd-a74e-d52516c48a67","Type":"ContainerDied","Data":"3b7df37e1f7780275f46a71a14fb434028457c4ff87b2e7206483e6b718a9bcf"}
Oct 31 14:02:59.890463 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.890329    4428 scope.go:122] "RemoveContainer" containerID="392317b6cb6029d6023b2ecfb6eebe05b0a075fb5f850a352c7866fd33d459f2"
Oct 31 14:02:59.890612 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.890468    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"
Oct 31 14:02:59.892586 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.892522    4428 generic.go:356] "Generic (PLEG): container finished" podID="ed8f2762-7f5a-4bf8-b40f-981f560b4e28" containerID="d6d64fd454f4dabfbaf1894863dba1a684b33b0af296a905ef07b9ec02fcd058" exitCode=0
Oct 31 14:02:59.892749 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.892632    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl" event={"ID":"ed8f2762-7f5a-4bf8-b40f-981f560b4e28","Type":"ContainerDied","Data":"d6d64fd454f4dabfbaf1894863dba1a684b33b0af296a905ef07b9ec02fcd058"}
Oct 31 14:02:59.892749 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.892679    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl" event={"ID":"ed8f2762-7f5a-4bf8-b40f-981f560b4e28","Type":"ContainerDied","Data":"e1f74ee07e1fa23f60e82fa16e6873edd89b3888c4ae93ec82e7acf314d127f4"}
Oct 31 14:02:59.892990 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.892763    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"
Oct 31 14:02:59.904182 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.903937    4428 generic.go:356] "Generic (PLEG): container finished" podID="4e024838-48ac-410b-8cce-cfb5f6ee6ef9" containerID="b1e840fee4bd4d02a1ecf5f42995f96d34f8a3dee4ad3c3219f4c5a1991198cd" exitCode=0
Oct 31 14:02:59.904182 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.904042    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt" event={"ID":"4e024838-48ac-410b-8cce-cfb5f6ee6ef9","Type":"ContainerDied","Data":"b1e840fee4bd4d02a1ecf5f42995f96d34f8a3dee4ad3c3219f4c5a1991198cd"}
Oct 31 14:02:59.904182 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.904119    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt" event={"ID":"4e024838-48ac-410b-8cce-cfb5f6ee6ef9","Type":"ContainerDied","Data":"022df114e5f77556c0872c3d31f4f5436580378c2d7048f751a712371bdee903"}
Oct 31 14:02:59.904182 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.904057    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"
Oct 31 14:02:59.908954 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.908856    4428 scope.go:122] "RemoveContainer" containerID="392317b6cb6029d6023b2ecfb6eebe05b0a075fb5f850a352c7866fd33d459f2"
Oct 31 14:02:59.909272 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:59.909241    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392317b6cb6029d6023b2ecfb6eebe05b0a075fb5f850a352c7866fd33d459f2\": container with ID starting with 392317b6cb6029d6023b2ecfb6eebe05b0a075fb5f850a352c7866fd33d459f2 not found: ID does not exist" containerID="392317b6cb6029d6023b2ecfb6eebe05b0a075fb5f850a352c7866fd33d459f2"
Oct 31 14:02:59.909371 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.909282    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392317b6cb6029d6023b2ecfb6eebe05b0a075fb5f850a352c7866fd33d459f2"} err="failed to get container status \"392317b6cb6029d6023b2ecfb6eebe05b0a075fb5f850a352c7866fd33d459f2\": rpc error: code = NotFound desc = could not find container \"392317b6cb6029d6023b2ecfb6eebe05b0a075fb5f850a352c7866fd33d459f2\": container with ID starting with 392317b6cb6029d6023b2ecfb6eebe05b0a075fb5f850a352c7866fd33d459f2 not found: ID does not exist"
Oct 31 14:02:59.909371 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.909305    4428 scope.go:122] "RemoveContainer" containerID="d6d64fd454f4dabfbaf1894863dba1a684b33b0af296a905ef07b9ec02fcd058"
Oct 31 14:02:59.923598 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.923579    4428 scope.go:122] "RemoveContainer" containerID="d6d64fd454f4dabfbaf1894863dba1a684b33b0af296a905ef07b9ec02fcd058"
Oct 31 14:02:59.924914 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:59.924886    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d64fd454f4dabfbaf1894863dba1a684b33b0af296a905ef07b9ec02fcd058\": container with ID starting with d6d64fd454f4dabfbaf1894863dba1a684b33b0af296a905ef07b9ec02fcd058 not found: ID does not exist" containerID="d6d64fd454f4dabfbaf1894863dba1a684b33b0af296a905ef07b9ec02fcd058"
Oct 31 14:02:59.924914 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.924923    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d64fd454f4dabfbaf1894863dba1a684b33b0af296a905ef07b9ec02fcd058"} err="failed to get container status \"d6d64fd454f4dabfbaf1894863dba1a684b33b0af296a905ef07b9ec02fcd058\": rpc error: code = NotFound desc = could not find container \"d6d64fd454f4dabfbaf1894863dba1a684b33b0af296a905ef07b9ec02fcd058\": container with ID starting with d6d64fd454f4dabfbaf1894863dba1a684b33b0af296a905ef07b9ec02fcd058 not found: ID does not exist"
Oct 31 14:02:59.924914 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.924948    4428 scope.go:122] "RemoveContainer" containerID="b1e840fee4bd4d02a1ecf5f42995f96d34f8a3dee4ad3c3219f4c5a1991198cd"
Oct 31 14:02:59.934833 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.934805    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"]
Oct 31 14:02:59.937635 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.937609    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-586469-6b5d89858-n9ndl"]
Oct 31 14:02:59.940247 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.940218    4428 scope.go:122] "RemoveContainer" containerID="b1e840fee4bd4d02a1ecf5f42995f96d34f8a3dee4ad3c3219f4c5a1991198cd"
Oct 31 14:02:59.940610 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:02:59.940585    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1e840fee4bd4d02a1ecf5f42995f96d34f8a3dee4ad3c3219f4c5a1991198cd\": container with ID starting with b1e840fee4bd4d02a1ecf5f42995f96d34f8a3dee4ad3c3219f4c5a1991198cd not found: ID does not exist" containerID="b1e840fee4bd4d02a1ecf5f42995f96d34f8a3dee4ad3c3219f4c5a1991198cd"
Oct 31 14:02:59.940716 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.940614    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1e840fee4bd4d02a1ecf5f42995f96d34f8a3dee4ad3c3219f4c5a1991198cd"} err="failed to get container status \"b1e840fee4bd4d02a1ecf5f42995f96d34f8a3dee4ad3c3219f4c5a1991198cd\": rpc error: code = NotFound desc = could not find container \"b1e840fee4bd4d02a1ecf5f42995f96d34f8a3dee4ad3c3219f4c5a1991198cd\": container with ID starting with b1e840fee4bd4d02a1ecf5f42995f96d34f8a3dee4ad3c3219f4c5a1991198cd not found: ID does not exist"
Oct 31 14:02:59.976136 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.976097    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e024838-48ac-410b-8cce-cfb5f6ee6ef9-kube-api-access-bw75r\" (UniqueName: \"kubernetes.io/projected/4e024838-48ac-410b-8cce-cfb5f6ee6ef9-kube-api-access-bw75r\") pod \"4e024838-48ac-410b-8cce-cfb5f6ee6ef9\" (UID: \"4e024838-48ac-410b-8cce-cfb5f6ee6ef9\") "
Oct 31 14:02:59.976303 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.976149    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c37feae-b089-45dd-a74e-d52516c48a67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c37feae-b089-45dd-a74e-d52516c48a67-plugins-config-volume\") pod \"1c37feae-b089-45dd-a74e-d52516c48a67\" (UID: \"1c37feae-b089-45dd-a74e-d52516c48a67\") "
Oct 31 14:02:59.976303 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.976205    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e024838-48ac-410b-8cce-cfb5f6ee6ef9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e024838-48ac-410b-8cce-cfb5f6ee6ef9-plugins-config-volume\") pod \"4e024838-48ac-410b-8cce-cfb5f6ee6ef9\" (UID: \"4e024838-48ac-410b-8cce-cfb5f6ee6ef9\") "
Oct 31 14:02:59.976303 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.976289    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c37feae-b089-45dd-a74e-d52516c48a67-kube-api-access-mp2vd\" (UniqueName: \"kubernetes.io/projected/1c37feae-b089-45dd-a74e-d52516c48a67-kube-api-access-mp2vd\") pod \"1c37feae-b089-45dd-a74e-d52516c48a67\" (UID: \"1c37feae-b089-45dd-a74e-d52516c48a67\") "
Oct 31 14:02:59.976637 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.976611    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e024838-48ac-410b-8cce-cfb5f6ee6ef9-plugins-config-volume" pod "4e024838-48ac-410b-8cce-cfb5f6ee6ef9" (UID: "4e024838-48ac-410b-8cce-cfb5f6ee6ef9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:59.976716 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.976610    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c37feae-b089-45dd-a74e-d52516c48a67-plugins-config-volume" pod "1c37feae-b089-45dd-a74e-d52516c48a67" (UID: "1c37feae-b089-45dd-a74e-d52516c48a67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:02:59.976760 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.976721    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rztr\" (UniqueName: \"kubernetes.io/projected/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-kube-api-access-7rztr\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:59.976760 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.976739    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e024838-48ac-410b-8cce-cfb5f6ee6ef9-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:59.976760 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.976750    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:59.976857 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.976760    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:59.976857 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.976769    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8f2762-7f5a-4bf8-b40f-981f560b4e28-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:59.976857 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.976777    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c37feae-b089-45dd-a74e-d52516c48a67-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:02:59.978503 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.978482    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c37feae-b089-45dd-a74e-d52516c48a67-kube-api-access-mp2vd" pod "1c37feae-b089-45dd-a74e-d52516c48a67" (UID: "1c37feae-b089-45dd-a74e-d52516c48a67"). InnerVolumeSpecName "kube-api-access-mp2vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:02:59.978577 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:02:59.978542    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e024838-48ac-410b-8cce-cfb5f6ee6ef9-kube-api-access-bw75r" pod "4e024838-48ac-410b-8cce-cfb5f6ee6ef9" (UID: "4e024838-48ac-410b-8cce-cfb5f6ee6ef9"). InnerVolumeSpecName "kube-api-access-bw75r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:00.032236 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:00.032204    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:00.077948 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:00.077913    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp2vd\" (UniqueName: \"kubernetes.io/projected/1c37feae-b089-45dd-a74e-d52516c48a67-kube-api-access-mp2vd\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:00.077948 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:00.077943    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw75r\" (UniqueName: \"kubernetes.io/projected/4e024838-48ac-410b-8cce-cfb5f6ee6ef9-kube-api-access-bw75r\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:00.218828 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:00.218801    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"]
Oct 31 14:03:00.225248 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:00.225220    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-586469-798774d6c5-pxksj"]
Oct 31 14:03:00.238053 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:00.238016    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"]
Oct 31 14:03:00.244534 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:00.244507    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-586469-7fd596fb78-28rgt"]
Oct 31 14:03:00.303803 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:00.303766    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:00.323921 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:00.322944    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"]
Oct 31 14:03:00.323921 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:00.323262    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj" podUID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" containerName="istio-proxy" containerID="cri-o://3b4362c55cd284b08313d0411fdca699da3a8f057ced98851c18b17bbaf45d1f" gracePeriod=30
Oct 31 14:03:00.512695 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:00.512659    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:00.754995 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:00.754958    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="004b6efe-f6d2-4c88-bb66-d31445c02d73" path="/var/lib/kubelet/pods/004b6efe-f6d2-4c88-bb66-d31445c02d73/volumes"
Oct 31 14:03:00.755389 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:00.755375    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c37feae-b089-45dd-a74e-d52516c48a67" path="/var/lib/kubelet/pods/1c37feae-b089-45dd-a74e-d52516c48a67/volumes"
Oct 31 14:03:00.755730 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:00.755716    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e024838-48ac-410b-8cce-cfb5f6ee6ef9" path="/var/lib/kubelet/pods/4e024838-48ac-410b-8cce-cfb5f6ee6ef9/volumes"
Oct 31 14:03:00.756467 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:00.756449    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed8f2762-7f5a-4bf8-b40f-981f560b4e28" path="/var/lib/kubelet/pods/ed8f2762-7f5a-4bf8-b40f-981f560b4e28/volumes"
Oct 31 14:03:01.098076 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:01.098041    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:02.090188 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:02.090145    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:02.661594 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:02.661555    4428 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:03:02.661765 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:02.661617    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj" podUID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:03:03.029742 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:03.029704    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:03.053074 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:03.053029    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w" podUID="b570ecdd-7dda-4021-966d-a36080ef5293" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:03:03.053290 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:03.053084    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp" podUID="7348227d-79bb-4bc8-b6ae-f4e318dcc703" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:03:03.413188 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:03.413100    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:04.088088 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:04.088037    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:04.622710 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:04.622661    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:04.785814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:04.785775    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:05.108622 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.108591    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:05.593645 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.593609    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 14:03:05.668184 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.668145    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-data\") pod \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") "
Oct 31 14:03:05.668712 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.668199    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istiod-ca-cert\") pod \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") "
Oct 31 14:03:05.668712 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.668280    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-envoy\") pod \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") "
Oct 31 14:03:05.668712 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.668334    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-kube-api-access-cml8j\" (UniqueName: \"kubernetes.io/projected/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-kube-api-access-cml8j\") pod \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") "
Oct 31 14:03:05.668712 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.668402    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-podinfo\") pod \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") "
Oct 31 14:03:05.668712 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.668443    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-token\" (UniqueName: \"kubernetes.io/projected/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-token\") pod \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") "
Oct 31 14:03:05.668712 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.668486    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-data" pod "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" (UID: "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:05.668712 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.668500    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-workload-socket\") pod \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") "
Oct 31 14:03:05.668712 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.668600    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istiod-ca-cert" pod "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" (UID: "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:03:05.668712 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.668599    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-credential-socket\") pod \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") "
Oct 31 14:03:05.669292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.668734    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-workload-socket" pod "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" (UID: "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:05.669292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.668658    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-workload-certs\") pod \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\" (UID: \"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6\") "
Oct 31 14:03:05.669292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.668765    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-credential-socket" pod "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" (UID: "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:05.669292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.669044    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-workload-certs" pod "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" (UID: "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:05.669292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.669248    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:05.669292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.669272    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:05.669292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.669287    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:05.669902 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.669302    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:05.669902 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.669316    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:05.671282 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.671256    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-envoy" pod "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" (UID: "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:05.671413 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.671276    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-token" pod "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" (UID: "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:05.671584 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.671549    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-podinfo" pod "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" (UID: "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:03:05.671665 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.671642    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-kube-api-access-cml8j" pod "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" (UID: "dfd4370a-9ed2-47aa-b9cf-8a6300944bd6"). InnerVolumeSpecName "kube-api-access-cml8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:05.770202 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.770168    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:05.770202 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.770195    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cml8j\" (UniqueName: \"kubernetes.io/projected/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-kube-api-access-cml8j\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:05.770202 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.770206    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:05.770455 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.770214    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:05.832865 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.832831    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:05.994703 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.994670    4428 generic.go:356] "Generic (PLEG): container finished" podID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" containerID="3b4362c55cd284b08313d0411fdca699da3a8f057ced98851c18b17bbaf45d1f" exitCode=0
Oct 31 14:03:05.994943 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.994718    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"
Oct 31 14:03:05.994943 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.994759    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj" event={"ID":"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6","Type":"ContainerDied","Data":"3b4362c55cd284b08313d0411fdca699da3a8f057ced98851c18b17bbaf45d1f"}
Oct 31 14:03:05.994943 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.994795    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:05.994943 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.994829    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj" event={"ID":"dfd4370a-9ed2-47aa-b9cf-8a6300944bd6","Type":"ContainerDied","Data":"20b7ff07f0c82c988de877083d42f953c75932037f2821e2c84090d2e24dd9b8"}
Oct 31 14:03:05.994943 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:05.994895    4428 scope.go:122] "RemoveContainer" containerID="3b4362c55cd284b08313d0411fdca699da3a8f057ced98851c18b17bbaf45d1f"
Oct 31 14:03:06.006623 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:06.006575    4428 scope.go:122] "RemoveContainer" containerID="4bc53ab41306a1cd1c8e0ce93d07ffc45d20dcc0f3d3676ef4e3185c5fcfbe05"
Oct 31 14:03:06.016562 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:06.016535    4428 scope.go:122] "RemoveContainer" containerID="3b4362c55cd284b08313d0411fdca699da3a8f057ced98851c18b17bbaf45d1f"
Oct 31 14:03:06.016962 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:03:06.016930    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b4362c55cd284b08313d0411fdca699da3a8f057ced98851c18b17bbaf45d1f\": container with ID starting with 3b4362c55cd284b08313d0411fdca699da3a8f057ced98851c18b17bbaf45d1f not found: ID does not exist" containerID="3b4362c55cd284b08313d0411fdca699da3a8f057ced98851c18b17bbaf45d1f"
Oct 31 14:03:06.017046 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:06.016976    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b4362c55cd284b08313d0411fdca699da3a8f057ced98851c18b17bbaf45d1f"} err="failed to get container status \"3b4362c55cd284b08313d0411fdca699da3a8f057ced98851c18b17bbaf45d1f\": rpc error: code = NotFound desc = could not find container \"3b4362c55cd284b08313d0411fdca699da3a8f057ced98851c18b17bbaf45d1f\": container with ID starting with 3b4362c55cd284b08313d0411fdca699da3a8f057ced98851c18b17bbaf45d1f not found: ID does not exist"
Oct 31 14:03:06.017046 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:06.017002    4428 scope.go:122] "RemoveContainer" containerID="4bc53ab41306a1cd1c8e0ce93d07ffc45d20dcc0f3d3676ef4e3185c5fcfbe05"
Oct 31 14:03:06.017310 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:03:06.017291    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc53ab41306a1cd1c8e0ce93d07ffc45d20dcc0f3d3676ef4e3185c5fcfbe05\": container with ID starting with 4bc53ab41306a1cd1c8e0ce93d07ffc45d20dcc0f3d3676ef4e3185c5fcfbe05 not found: ID does not exist" containerID="4bc53ab41306a1cd1c8e0ce93d07ffc45d20dcc0f3d3676ef4e3185c5fcfbe05"
Oct 31 14:03:06.017408 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:06.017316    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc53ab41306a1cd1c8e0ce93d07ffc45d20dcc0f3d3676ef4e3185c5fcfbe05"} err="failed to get container status \"4bc53ab41306a1cd1c8e0ce93d07ffc45d20dcc0f3d3676ef4e3185c5fcfbe05\": rpc error: code = NotFound desc = could not find container \"4bc53ab41306a1cd1c8e0ce93d07ffc45d20dcc0f3d3676ef4e3185c5fcfbe05\": container with ID starting with 4bc53ab41306a1cd1c8e0ce93d07ffc45d20dcc0f3d3676ef4e3185c5fcfbe05 not found: ID does not exist"
Oct 31 14:03:06.023097 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:06.023036    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"]
Oct 31 14:03:06.028056 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:06.028012    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-gpdxj"]
Oct 31 14:03:06.074538 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:06.074500    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"]
Oct 31 14:03:06.080937 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:06.080895    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aabeef-t5dkq"]
Oct 31 14:03:06.158957 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:06.158886    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:06.313458 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:06.313373    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:06.555999 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:06.555970    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:06.726302 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:06.726269    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:06.754856 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:06.754823    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd6dba80-2e73-4b66-809a-3383e52fd9fb" path="/var/lib/kubelet/pods/bd6dba80-2e73-4b66-809a-3383e52fd9fb/volumes"
Oct 31 14:03:06.755367 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:06.755351    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" path="/var/lib/kubelet/pods/dfd4370a-9ed2-47aa-b9cf-8a6300944bd6/volumes"
Oct 31 14:03:07.285849 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:07.284639    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:07.656049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:07.656021    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"
Oct 31 14:03:07.687644 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:07.687603    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"
Oct 31 14:03:07.689506 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:07.689471    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7348227d-79bb-4bc8-b6ae-f4e318dcc703-kube-api-access-rv4vp\" (UniqueName: \"kubernetes.io/projected/7348227d-79bb-4bc8-b6ae-f4e318dcc703-kube-api-access-rv4vp\") pod \"7348227d-79bb-4bc8-b6ae-f4e318dcc703\" (UID: \"7348227d-79bb-4bc8-b6ae-f4e318dcc703\") "
Oct 31 14:03:07.689691 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:07.689641    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7348227d-79bb-4bc8-b6ae-f4e318dcc703-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7348227d-79bb-4bc8-b6ae-f4e318dcc703-plugins-config-volume\") pod \"7348227d-79bb-4bc8-b6ae-f4e318dcc703\" (UID: \"7348227d-79bb-4bc8-b6ae-f4e318dcc703\") "
Oct 31 14:03:07.690083 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:07.690060    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7348227d-79bb-4bc8-b6ae-f4e318dcc703-plugins-config-volume" pod "7348227d-79bb-4bc8-b6ae-f4e318dcc703" (UID: "7348227d-79bb-4bc8-b6ae-f4e318dcc703"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:03:07.692152 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:07.692127    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7348227d-79bb-4bc8-b6ae-f4e318dcc703-kube-api-access-rv4vp" pod "7348227d-79bb-4bc8-b6ae-f4e318dcc703" (UID: "7348227d-79bb-4bc8-b6ae-f4e318dcc703"). InnerVolumeSpecName "kube-api-access-rv4vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:07.790689 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:07.790651    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b570ecdd-7dda-4021-966d-a36080ef5293-kube-api-access-k2q5t\" (UniqueName: \"kubernetes.io/projected/b570ecdd-7dda-4021-966d-a36080ef5293-kube-api-access-k2q5t\") pod \"b570ecdd-7dda-4021-966d-a36080ef5293\" (UID: \"b570ecdd-7dda-4021-966d-a36080ef5293\") "
Oct 31 14:03:07.791170 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:07.790760    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b570ecdd-7dda-4021-966d-a36080ef5293-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b570ecdd-7dda-4021-966d-a36080ef5293-plugins-config-volume\") pod \"b570ecdd-7dda-4021-966d-a36080ef5293\" (UID: \"b570ecdd-7dda-4021-966d-a36080ef5293\") "
Oct 31 14:03:07.791170 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:07.791085    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv4vp\" (UniqueName: \"kubernetes.io/projected/7348227d-79bb-4bc8-b6ae-f4e318dcc703-kube-api-access-rv4vp\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:07.791170 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:07.791097    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7348227d-79bb-4bc8-b6ae-f4e318dcc703-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:07.791170 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:07.791153    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b570ecdd-7dda-4021-966d-a36080ef5293-plugins-config-volume" pod "b570ecdd-7dda-4021-966d-a36080ef5293" (UID: "b570ecdd-7dda-4021-966d-a36080ef5293"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:03:07.793152 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:07.793129    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b570ecdd-7dda-4021-966d-a36080ef5293-kube-api-access-k2q5t" pod "b570ecdd-7dda-4021-966d-a36080ef5293" (UID: "b570ecdd-7dda-4021-966d-a36080ef5293"). InnerVolumeSpecName "kube-api-access-k2q5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:07.891743 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:07.891650    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2q5t\" (UniqueName: \"kubernetes.io/projected/b570ecdd-7dda-4021-966d-a36080ef5293-kube-api-access-k2q5t\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:07.891743 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:07.891686    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b570ecdd-7dda-4021-966d-a36080ef5293-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:08.012266 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.012230    4428 generic.go:356] "Generic (PLEG): container finished" podID="7348227d-79bb-4bc8-b6ae-f4e318dcc703" containerID="25bc270a2a63194bfe9a59e2adcef830df0af366088125fd7a6d2dbd36d38c27" exitCode=0
Oct 31 14:03:08.012475 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.012282    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"
Oct 31 14:03:08.012475 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.012312    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp" event={"ID":"7348227d-79bb-4bc8-b6ae-f4e318dcc703","Type":"ContainerDied","Data":"25bc270a2a63194bfe9a59e2adcef830df0af366088125fd7a6d2dbd36d38c27"}
Oct 31 14:03:08.012475 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.012379    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp" event={"ID":"7348227d-79bb-4bc8-b6ae-f4e318dcc703","Type":"ContainerDied","Data":"370c3b8f8b496f2d21e7c7280f23424b80657dbbc6bd48a8493caf7f1b11004e"}
Oct 31 14:03:08.012475 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.012409    4428 scope.go:122] "RemoveContainer" containerID="25bc270a2a63194bfe9a59e2adcef830df0af366088125fd7a6d2dbd36d38c27"
Oct 31 14:03:08.014074 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.014048    4428 generic.go:356] "Generic (PLEG): container finished" podID="b570ecdd-7dda-4021-966d-a36080ef5293" containerID="9aecfa60efd946312185571004b7166b43676f198ea380eab6cc580418e76dd7" exitCode=0
Oct 31 14:03:08.014212 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.014100    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"
Oct 31 14:03:08.014212 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.014124    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w" event={"ID":"b570ecdd-7dda-4021-966d-a36080ef5293","Type":"ContainerDied","Data":"9aecfa60efd946312185571004b7166b43676f198ea380eab6cc580418e76dd7"}
Oct 31 14:03:08.014212 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.014163    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w" event={"ID":"b570ecdd-7dda-4021-966d-a36080ef5293","Type":"ContainerDied","Data":"ebe130c912b8c994155ed78b4b8a9c6573323beb07de02e9ddb648d56872714f"}
Oct 31 14:03:08.023421 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.023403    4428 scope.go:122] "RemoveContainer" containerID="25bc270a2a63194bfe9a59e2adcef830df0af366088125fd7a6d2dbd36d38c27"
Oct 31 14:03:08.023791 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:03:08.023709    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25bc270a2a63194bfe9a59e2adcef830df0af366088125fd7a6d2dbd36d38c27\": container with ID starting with 25bc270a2a63194bfe9a59e2adcef830df0af366088125fd7a6d2dbd36d38c27 not found: ID does not exist" containerID="25bc270a2a63194bfe9a59e2adcef830df0af366088125fd7a6d2dbd36d38c27"
Oct 31 14:03:08.023791 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.023744    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25bc270a2a63194bfe9a59e2adcef830df0af366088125fd7a6d2dbd36d38c27"} err="failed to get container status \"25bc270a2a63194bfe9a59e2adcef830df0af366088125fd7a6d2dbd36d38c27\": rpc error: code = NotFound desc = could not find container \"25bc270a2a63194bfe9a59e2adcef830df0af366088125fd7a6d2dbd36d38c27\": container with ID starting with 25bc270a2a63194bfe9a59e2adcef830df0af366088125fd7a6d2dbd36d38c27 not found: ID does not exist"
Oct 31 14:03:08.023791 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.023768    4428 scope.go:122] "RemoveContainer" containerID="9aecfa60efd946312185571004b7166b43676f198ea380eab6cc580418e76dd7"
Oct 31 14:03:08.033534 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.033516    4428 scope.go:122] "RemoveContainer" containerID="9aecfa60efd946312185571004b7166b43676f198ea380eab6cc580418e76dd7"
Oct 31 14:03:08.033799 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:03:08.033780    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aecfa60efd946312185571004b7166b43676f198ea380eab6cc580418e76dd7\": container with ID starting with 9aecfa60efd946312185571004b7166b43676f198ea380eab6cc580418e76dd7 not found: ID does not exist" containerID="9aecfa60efd946312185571004b7166b43676f198ea380eab6cc580418e76dd7"
Oct 31 14:03:08.033847 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.033806    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aecfa60efd946312185571004b7166b43676f198ea380eab6cc580418e76dd7"} err="failed to get container status \"9aecfa60efd946312185571004b7166b43676f198ea380eab6cc580418e76dd7\": rpc error: code = NotFound desc = could not find container \"9aecfa60efd946312185571004b7166b43676f198ea380eab6cc580418e76dd7\": container with ID starting with 9aecfa60efd946312185571004b7166b43676f198ea380eab6cc580418e76dd7 not found: ID does not exist"
Oct 31 14:03:08.042099 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.042073    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"]
Oct 31 14:03:08.044704 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.044683    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aabeef-7fbf76c96c-cx8wp"]
Oct 31 14:03:08.057946 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.057924    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"]
Oct 31 14:03:08.063729 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.063707    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aabeef-585b7b8c4c-gbl4w"]
Oct 31 14:03:08.154451 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.154304    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:08.465319 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.465280    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:08.680884 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.680819    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:08.757904 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.757817    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7348227d-79bb-4bc8-b6ae-f4e318dcc703" path="/var/lib/kubelet/pods/7348227d-79bb-4bc8-b6ae-f4e318dcc703/volumes"
Oct 31 14:03:08.758376 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.758356    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b570ecdd-7dda-4021-966d-a36080ef5293" path="/var/lib/kubelet/pods/b570ecdd-7dda-4021-966d-a36080ef5293/volumes"
Oct 31 14:03:08.772739 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.772708    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"]
Oct 31 14:03:08.773007 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.772983    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m" podUID="8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" containerName="istio-proxy" containerID="cri-o://556e167c37c8a59ec7c4eea8827eb5c1951251d2acbb5cb92ae5169dcc69bfe8" gracePeriod=30
Oct 31 14:03:08.919888 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:08.919842    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:09.273067 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:09.273022    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:10.111553 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:10.111514    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:11.011350 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:11.011288    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:12.033926 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:12.033891    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:12.817560 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:12.817512    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:12.978451 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:12.978410    4428 patch_prober.go:29] interesting pod/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:03:12.978612 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:12.978482    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m" podUID="8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:03:13.392443 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.392405    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:13.714759 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.714686    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:13.793020 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.792986    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"]
Oct 31 14:03:13.793542 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793526    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4e024838-48ac-410b-8cce-cfb5f6ee6ef9" containerName="payload-processing"
Oct 31 14:03:13.793623 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793543    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e024838-48ac-410b-8cce-cfb5f6ee6ef9" containerName="payload-processing"
Oct 31 14:03:13.793623 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793554    4428 cpu_manager.go:414] "containerMap: removing container" podUID="004b6efe-f6d2-4c88-bb66-d31445c02d73" containerName="revoke-keys"
Oct 31 14:03:13.793623 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793559    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004b6efe-f6d2-4c88-bb66-d31445c02d73" containerName="revoke-keys"
Oct 31 14:03:13.793623 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793571    4428 cpu_manager.go:414] "containerMap: removing container" podUID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" containerName="istio-proxy"
Oct 31 14:03:13.793623 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793576    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" containerName="istio-proxy"
Oct 31 14:03:13.793623 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793585    4428 cpu_manager.go:414] "containerMap: removing container" podUID="1c37feae-b089-45dd-a74e-d52516c48a67" containerName="payload-pre-processing"
Oct 31 14:03:13.793623 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793590    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c37feae-b089-45dd-a74e-d52516c48a67" containerName="payload-pre-processing"
Oct 31 14:03:13.793623 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793599    4428 cpu_manager.go:414] "containerMap: removing container" podUID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" containerName="istio-proxy"
Oct 31 14:03:13.793623 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793604    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" containerName="istio-proxy"
Oct 31 14:03:13.793623 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793620    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b570ecdd-7dda-4021-966d-a36080ef5293" containerName="payload-pre-processing"
Oct 31 14:03:13.793623 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793624    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b570ecdd-7dda-4021-966d-a36080ef5293" containerName="payload-pre-processing"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793631    4428 cpu_manager.go:414] "containerMap: removing container" podUID="07a6debd-03b9-4487-9d39-b9fa19078ea9" containerName="revoke-keys"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793635    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a6debd-03b9-4487-9d39-b9fa19078ea9" containerName="revoke-keys"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793643    4428 cpu_manager.go:414] "containerMap: removing container" podUID="7348227d-79bb-4bc8-b6ae-f4e318dcc703" containerName="payload-processing"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793647    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7348227d-79bb-4bc8-b6ae-f4e318dcc703" containerName="payload-processing"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793654    4428 cpu_manager.go:414] "containerMap: removing container" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793659    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793666    4428 cpu_manager.go:414] "containerMap: removing container" podUID="ee893209-7839-4a39-8aaa-17dc0f90d8d7" containerName="payload-pre-processing"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793670    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee893209-7839-4a39-8aaa-17dc0f90d8d7" containerName="payload-pre-processing"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793684    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793688    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793695    4428 cpu_manager.go:414] "containerMap: removing container" podUID="ed8f2762-7f5a-4bf8-b40f-981f560b4e28" containerName="maas-api"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793700    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed8f2762-7f5a-4bf8-b40f-981f560b4e28" containerName="maas-api"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793763    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee893209-7839-4a39-8aaa-17dc0f90d8d7" containerName="payload-pre-processing"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793774    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b950b997-7653-480d-b120-31dc384832d0" containerName="main"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793780    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed8f2762-7f5a-4bf8-b40f-981f560b4e28" containerName="maas-api"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793788    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e024838-48ac-410b-8cce-cfb5f6ee6ef9" containerName="payload-processing"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793793    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="004b6efe-f6d2-4c88-bb66-d31445c02d73" containerName="revoke-keys"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793798    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" containerName="istio-proxy"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793804    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c37feae-b089-45dd-a74e-d52516c48a67" containerName="payload-pre-processing"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793811    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd4370a-9ed2-47aa-b9cf-8a6300944bd6" containerName="istio-proxy"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793818    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a6debd-03b9-4487-9d39-b9fa19078ea9" containerName="revoke-keys"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793824    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b570ecdd-7dda-4021-966d-a36080ef5293" containerName="payload-pre-processing"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793830    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="7348227d-79bb-4bc8-b6ae-f4e318dcc703" containerName="payload-processing"
Oct 31 14:03:13.794049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.793835    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6a586d-e1a2-4965-b0fc-89e2286d7298" containerName="main"
Oct 31 14:03:13.797046 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.797029    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"
Oct 31 14:03:13.803795 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.803770    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:03:13.803959 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.803816    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:03:13.807927 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.807812    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"]
Oct 31 14:03:13.851121 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.851086    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbx4h\" (UniqueName: \"kubernetes.io/projected/f58d4555-ef4a-4a23-9438-8fad9bae554e-kube-api-access-cbx4h\") pod \"maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd\" (UID: \"f58d4555-ef4a-4a23-9438-8fad9bae554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"
Oct 31 14:03:13.851303 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.851129    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58d4555-ef4a-4a23-9438-8fad9bae554e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd\" (UID: \"f58d4555-ef4a-4a23-9438-8fad9bae554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"
Oct 31 14:03:13.851303 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.851280    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58d4555-ef4a-4a23-9438-8fad9bae554e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd\" (UID: \"f58d4555-ef4a-4a23-9438-8fad9bae554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"
Oct 31 14:03:13.954560 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.954496    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58d4555-ef4a-4a23-9438-8fad9bae554e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd\" (UID: \"f58d4555-ef4a-4a23-9438-8fad9bae554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"
Oct 31 14:03:13.954780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.954660    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbx4h\" (UniqueName: \"kubernetes.io/projected/f58d4555-ef4a-4a23-9438-8fad9bae554e-kube-api-access-cbx4h\") pod \"maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd\" (UID: \"f58d4555-ef4a-4a23-9438-8fad9bae554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"
Oct 31 14:03:13.954780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.954700    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58d4555-ef4a-4a23-9438-8fad9bae554e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd\" (UID: \"f58d4555-ef4a-4a23-9438-8fad9bae554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"
Oct 31 14:03:13.955424 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.955376    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58d4555-ef4a-4a23-9438-8fad9bae554e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd\" (UID: \"f58d4555-ef4a-4a23-9438-8fad9bae554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"
Oct 31 14:03:13.958406 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.958379    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58d4555-ef4a-4a23-9438-8fad9bae554e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd\" (UID: \"f58d4555-ef4a-4a23-9438-8fad9bae554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"
Oct 31 14:03:13.974182 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:13.974105    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbx4h\" (UniqueName: \"kubernetes.io/projected/f58d4555-ef4a-4a23-9438-8fad9bae554e-kube-api-access-cbx4h\") pod \"maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd\" (UID: \"f58d4555-ef4a-4a23-9438-8fad9bae554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"
Oct 31 14:03:14.040830 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.040802    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:03:14.057515 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.057486    4428 generic.go:356] "Generic (PLEG): container finished" podID="8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" containerID="556e167c37c8a59ec7c4eea8827eb5c1951251d2acbb5cb92ae5169dcc69bfe8" exitCode=0
Oct 31 14:03:14.057686 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.057537    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"
Oct 31 14:03:14.057686 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.057571    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m" event={"ID":"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21","Type":"ContainerDied","Data":"556e167c37c8a59ec7c4eea8827eb5c1951251d2acbb5cb92ae5169dcc69bfe8"}
Oct 31 14:03:14.057686 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.057631    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m" event={"ID":"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21","Type":"ContainerDied","Data":"a588ecbf8687c60fba4e1383ad0763ca8a3624c12a358d02ae50c985f8063ce4"}
Oct 31 14:03:14.057686 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.057658    4428 scope.go:122] "RemoveContainer" containerID="556e167c37c8a59ec7c4eea8827eb5c1951251d2acbb5cb92ae5169dcc69bfe8"
Oct 31 14:03:14.068253 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.068219    4428 scope.go:122] "RemoveContainer" containerID="d57653db8e69ce08b52d338fd62bdce1a79e7e92f409d88643ea1a6dbd738930"
Oct 31 14:03:14.080018 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.079993    4428 scope.go:122] "RemoveContainer" containerID="556e167c37c8a59ec7c4eea8827eb5c1951251d2acbb5cb92ae5169dcc69bfe8"
Oct 31 14:03:14.080428 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:03:14.080399    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556e167c37c8a59ec7c4eea8827eb5c1951251d2acbb5cb92ae5169dcc69bfe8\": container with ID starting with 556e167c37c8a59ec7c4eea8827eb5c1951251d2acbb5cb92ae5169dcc69bfe8 not found: ID does not exist" containerID="556e167c37c8a59ec7c4eea8827eb5c1951251d2acbb5cb92ae5169dcc69bfe8"
Oct 31 14:03:14.080499 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.080441    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556e167c37c8a59ec7c4eea8827eb5c1951251d2acbb5cb92ae5169dcc69bfe8"} err="failed to get container status \"556e167c37c8a59ec7c4eea8827eb5c1951251d2acbb5cb92ae5169dcc69bfe8\": rpc error: code = NotFound desc = could not find container \"556e167c37c8a59ec7c4eea8827eb5c1951251d2acbb5cb92ae5169dcc69bfe8\": container with ID starting with 556e167c37c8a59ec7c4eea8827eb5c1951251d2acbb5cb92ae5169dcc69bfe8 not found: ID does not exist"
Oct 31 14:03:14.080499 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.080467    4428 scope.go:122] "RemoveContainer" containerID="d57653db8e69ce08b52d338fd62bdce1a79e7e92f409d88643ea1a6dbd738930"
Oct 31 14:03:14.080801 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:03:14.080774    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57653db8e69ce08b52d338fd62bdce1a79e7e92f409d88643ea1a6dbd738930\": container with ID starting with d57653db8e69ce08b52d338fd62bdce1a79e7e92f409d88643ea1a6dbd738930 not found: ID does not exist" containerID="d57653db8e69ce08b52d338fd62bdce1a79e7e92f409d88643ea1a6dbd738930"
Oct 31 14:03:14.080922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.080811    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57653db8e69ce08b52d338fd62bdce1a79e7e92f409d88643ea1a6dbd738930"} err="failed to get container status \"d57653db8e69ce08b52d338fd62bdce1a79e7e92f409d88643ea1a6dbd738930\": rpc error: code = NotFound desc = could not find container \"d57653db8e69ce08b52d338fd62bdce1a79e7e92f409d88643ea1a6dbd738930\": container with ID starting with d57653db8e69ce08b52d338fd62bdce1a79e7e92f409d88643ea1a6dbd738930 not found: ID does not exist"
Oct 31 14:03:14.156608 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.156576    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-token\" (UniqueName: \"kubernetes.io/projected/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-token\") pod \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") "
Oct 31 14:03:14.156798 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.156630    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-workload-socket\") pod \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") "
Oct 31 14:03:14.156798 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.156657    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-workload-certs\") pod \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") "
Oct 31 14:03:14.156798 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.156686    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-data\") pod \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") "
Oct 31 14:03:14.156798 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.156713    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-podinfo\") pod \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") "
Oct 31 14:03:14.156798 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.156746    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-credential-socket\") pod \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") "
Oct 31 14:03:14.157179 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.156796    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istiod-ca-cert\") pod \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") "
Oct 31 14:03:14.157179 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.156915    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-envoy\") pod \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") "
Oct 31 14:03:14.157179 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.156949    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-kube-api-access-dhxwg\" (UniqueName: \"kubernetes.io/projected/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-kube-api-access-dhxwg\") pod \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\" (UID: \"8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21\") "
Oct 31 14:03:14.157179 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.157009    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-workload-socket" pod "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" (UID: "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:14.157179 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.157028    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-workload-certs" pod "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" (UID: "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:14.157179 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.157042    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-data" pod "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" (UID: "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:14.157576 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.157300    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istiod-ca-cert" pod "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" (UID: "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:03:14.157576 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.157309    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-credential-socket" pod "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" (UID: "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:14.157576 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.157458    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:14.157576 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.157481    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:14.157576 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.157495    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:14.157576 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.157509    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:14.157576 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.157523    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:14.159167 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.159139    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-envoy" pod "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" (UID: "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:14.159310 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.159173    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-token" pod "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" (UID: "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:14.159310 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.159195    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-podinfo" pod "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" (UID: "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:03:14.159451 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.159411    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-kube-api-access-dhxwg" pod "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" (UID: "8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21"). InnerVolumeSpecName "kube-api-access-dhxwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:14.161818 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.161793    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"
Oct 31 14:03:14.258859 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.258769    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:14.258859 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.258819    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:14.258859 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.258836    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhxwg\" (UniqueName: \"kubernetes.io/projected/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-kube-api-access-dhxwg\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:14.258859 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.258852    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:14.383192 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.383158    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"]
Oct 31 14:03:14.387689 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.387662    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-586469-openshift-default-6cb999f45f-7q54m"]
Oct 31 14:03:14.447837 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.447803    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"]
Oct 31 14:03:14.450096 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:03:14.450062    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58d4555_ef4a_4a23_9438_8fad9bae554e.slice/crio-597f8c233cedbe5d1b81aea48d3e47f2b6e182a5ea20b39d44e3796286ea858e WatchSource:0}: Error finding container 597f8c233cedbe5d1b81aea48d3e47f2b6e182a5ea20b39d44e3796286ea858e: Status 404 returned error can't find the container with id 597f8c233cedbe5d1b81aea48d3e47f2b6e182a5ea20b39d44e3796286ea858e
Oct 31 14:03:14.546528 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.546431    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:14.768675 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.768643    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:14.784075 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.784031    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" path="/var/lib/kubelet/pods/8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21/volumes"
Oct 31 14:03:14.931402 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:14.931361    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:15.090397 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:15.090299    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd" event={"ID":"f58d4555-ef4a-4a23-9438-8fad9bae554e","Type":"ContainerStarted","Data":"c73a0f2f98ffd03940fc725cffbad8e9564bbc0f533f332d5419f7c3baa15b72"}
Oct 31 14:03:15.090397 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:15.090372    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd" event={"ID":"f58d4555-ef4a-4a23-9438-8fad9bae554e","Type":"ContainerStarted","Data":"597f8c233cedbe5d1b81aea48d3e47f2b6e182a5ea20b39d44e3796286ea858e"}
Oct 31 14:03:15.098293 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:15.098259    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:15.112571 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:15.112513    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd" podStartSLOduration=2.112496175 podStartE2EDuration="2.112496175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:03:15.110248937 +0000 UTC m=+3592.508597562" watchObservedRunningTime="2026-10-31 14:03:15.112496175 +0000 UTC m=+3592.510844799"
Oct 31 14:03:15.214892 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:15.214828    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"]
Oct 31 14:03:15.215167 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:15.215136    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj" podUID="d5e6e17a-543e-4373-ad93-e0ed60c74334" containerName="istio-proxy" containerID="cri-o://b2c4e91a017758f0511a49cac163f50e1fac4d2a41a2b8fbe9ae97f034bc7a7f" gracePeriod=30
Oct 31 14:03:15.263162 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:15.263126    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:15.462257 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:15.462226    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:15.878704 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:15.878603    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:16.051816 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:16.051778    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:16.098500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:16.098464    4428 generic.go:356] "Generic (PLEG): container finished" podID="f58d4555-ef4a-4a23-9438-8fad9bae554e" containerID="c73a0f2f98ffd03940fc725cffbad8e9564bbc0f533f332d5419f7c3baa15b72" exitCode=0
Oct 31 14:03:16.098695 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:16.098502    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd" event={"ID":"f58d4555-ef4a-4a23-9438-8fad9bae554e","Type":"ContainerDied","Data":"c73a0f2f98ffd03940fc725cffbad8e9564bbc0f533f332d5419f7c3baa15b72"}
Oct 31 14:03:16.497037 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:16.497002    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:16.704606 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:16.704567    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:17.257784 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:17.257759    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"
Oct 31 14:03:17.303954 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:17.303922    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58d4555-ef4a-4a23-9438-8fad9bae554e-kube-api-access-cbx4h\" (UniqueName: \"kubernetes.io/projected/f58d4555-ef4a-4a23-9438-8fad9bae554e-kube-api-access-cbx4h\") pod \"f58d4555-ef4a-4a23-9438-8fad9bae554e\" (UID: \"f58d4555-ef4a-4a23-9438-8fad9bae554e\") "
Oct 31 14:03:17.304168 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:17.303977    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58d4555-ef4a-4a23-9438-8fad9bae554e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58d4555-ef4a-4a23-9438-8fad9bae554e-maas-api-openshift-ca\") pod \"f58d4555-ef4a-4a23-9438-8fad9bae554e\" (UID: \"f58d4555-ef4a-4a23-9438-8fad9bae554e\") "
Oct 31 14:03:17.304168 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:17.304018    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58d4555-ef4a-4a23-9438-8fad9bae554e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58d4555-ef4a-4a23-9438-8fad9bae554e-maas-api-xks-ca\") pod \"f58d4555-ef4a-4a23-9438-8fad9bae554e\" (UID: \"f58d4555-ef4a-4a23-9438-8fad9bae554e\") "
Oct 31 14:03:17.304442 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:17.304406    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58d4555-ef4a-4a23-9438-8fad9bae554e-maas-api-openshift-ca" pod "f58d4555-ef4a-4a23-9438-8fad9bae554e" (UID: "f58d4555-ef4a-4a23-9438-8fad9bae554e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:03:17.306508 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:17.306481    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58d4555-ef4a-4a23-9438-8fad9bae554e-maas-api-xks-ca" pod "f58d4555-ef4a-4a23-9438-8fad9bae554e" (UID: "f58d4555-ef4a-4a23-9438-8fad9bae554e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:03:17.306508 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:17.306498    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58d4555-ef4a-4a23-9438-8fad9bae554e-kube-api-access-cbx4h" pod "f58d4555-ef4a-4a23-9438-8fad9bae554e" (UID: "f58d4555-ef4a-4a23-9438-8fad9bae554e"). InnerVolumeSpecName "kube-api-access-cbx4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:17.405815 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:17.405729    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbx4h\" (UniqueName: \"kubernetes.io/projected/f58d4555-ef4a-4a23-9438-8fad9bae554e-kube-api-access-cbx4h\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:17.405815 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:17.405759    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58d4555-ef4a-4a23-9438-8fad9bae554e-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:17.405815 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:17.405770    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58d4555-ef4a-4a23-9438-8fad9bae554e-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:17.649169 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:17.649133    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:17.805220 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:17.805183    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:18.113428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:18.113331    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd" event={"ID":"f58d4555-ef4a-4a23-9438-8fad9bae554e","Type":"ContainerDied","Data":"597f8c233cedbe5d1b81aea48d3e47f2b6e182a5ea20b39d44e3796286ea858e"}
Oct 31 14:03:18.113428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:18.113362    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"
Oct 31 14:03:18.113428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:18.113393    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="597f8c233cedbe5d1b81aea48d3e47f2b6e182a5ea20b39d44e3796286ea858e"
Oct 31 14:03:18.390635 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:18.390547    4428 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:03:18.390635 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:18.390610    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj" podUID="d5e6e17a-543e-4373-ad93-e0ed60c74334" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:03:18.682192 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:18.682101    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:18.745028 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:03:18.744993    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:03:19.409741 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:19.409700    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:20.202051 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.202012    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:20.473997 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.473972    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 14:03:20.536742 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.536706    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e6e17a-543e-4373-ad93-e0ed60c74334-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e6e17a-543e-4373-ad93-e0ed60c74334-istiod-ca-cert\") pod \"d5e6e17a-543e-4373-ad93-e0ed60c74334\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") "
Oct 31 14:03:20.536742 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.536748    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-workload-certs\") pod \"d5e6e17a-543e-4373-ad93-e0ed60c74334\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") "
Oct 31 14:03:20.537069 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.536789    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e6e17a-543e-4373-ad93-e0ed60c74334-kube-api-access-dsxvp\" (UniqueName: \"kubernetes.io/projected/d5e6e17a-543e-4373-ad93-e0ed60c74334-kube-api-access-dsxvp\") pod \"d5e6e17a-543e-4373-ad93-e0ed60c74334\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") "
Oct 31 14:03:20.537069 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.536899    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-credential-socket\") pod \"d5e6e17a-543e-4373-ad93-e0ed60c74334\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") "
Oct 31 14:03:20.537069 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.536928    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-podinfo\") pod \"d5e6e17a-543e-4373-ad93-e0ed60c74334\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") "
Oct 31 14:03:20.537069 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.536977    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-token\" (UniqueName: \"kubernetes.io/projected/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-token\") pod \"d5e6e17a-543e-4373-ad93-e0ed60c74334\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") "
Oct 31 14:03:20.537314 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.537103    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-workload-certs" pod "d5e6e17a-543e-4373-ad93-e0ed60c74334" (UID: "d5e6e17a-543e-4373-ad93-e0ed60c74334"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:20.537314 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.537162    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e6e17a-543e-4373-ad93-e0ed60c74334-istiod-ca-cert" pod "d5e6e17a-543e-4373-ad93-e0ed60c74334" (UID: "d5e6e17a-543e-4373-ad93-e0ed60c74334"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:03:20.537314 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.537290    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-credential-socket" pod "d5e6e17a-543e-4373-ad93-e0ed60c74334" (UID: "d5e6e17a-543e-4373-ad93-e0ed60c74334"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:20.537486 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.537422    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-workload-socket" pod "d5e6e17a-543e-4373-ad93-e0ed60c74334" (UID: "d5e6e17a-543e-4373-ad93-e0ed60c74334"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:20.537612 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.537581    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-workload-socket\") pod \"d5e6e17a-543e-4373-ad93-e0ed60c74334\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") "
Oct 31 14:03:20.537777 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.537656    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-data\") pod \"d5e6e17a-543e-4373-ad93-e0ed60c74334\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") "
Oct 31 14:03:20.538241 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.538203    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-data" pod "d5e6e17a-543e-4373-ad93-e0ed60c74334" (UID: "d5e6e17a-543e-4373-ad93-e0ed60c74334"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:20.538456 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.538433    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-envoy\") pod \"d5e6e17a-543e-4373-ad93-e0ed60c74334\" (UID: \"d5e6e17a-543e-4373-ad93-e0ed60c74334\") "
Oct 31 14:03:20.538960 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.538935    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:20.539128 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.539099    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:20.539292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.539129    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:20.539292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.539148    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e6e17a-543e-4373-ad93-e0ed60c74334-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:20.539292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.539163    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:20.539919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.539833    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-token" pod "d5e6e17a-543e-4373-ad93-e0ed60c74334" (UID: "d5e6e17a-543e-4373-ad93-e0ed60c74334"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:20.540016 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.539977    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e6e17a-543e-4373-ad93-e0ed60c74334-kube-api-access-dsxvp" pod "d5e6e17a-543e-4373-ad93-e0ed60c74334" (UID: "d5e6e17a-543e-4373-ad93-e0ed60c74334"). InnerVolumeSpecName "kube-api-access-dsxvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:20.541970 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.541946    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-podinfo" pod "d5e6e17a-543e-4373-ad93-e0ed60c74334" (UID: "d5e6e17a-543e-4373-ad93-e0ed60c74334"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:03:20.542109 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.542085    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-envoy" pod "d5e6e17a-543e-4373-ad93-e0ed60c74334" (UID: "d5e6e17a-543e-4373-ad93-e0ed60c74334"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:20.640517 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.640478    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsxvp\" (UniqueName: \"kubernetes.io/projected/d5e6e17a-543e-4373-ad93-e0ed60c74334-kube-api-access-dsxvp\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:20.640517 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.640512    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:20.640517 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.640523    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:20.640787 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:20.640532    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e6e17a-543e-4373-ad93-e0ed60c74334-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:21.047440 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:21.047324    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:21.134072 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:21.134031    4428 generic.go:356] "Generic (PLEG): container finished" podID="d5e6e17a-543e-4373-ad93-e0ed60c74334" containerID="b2c4e91a017758f0511a49cac163f50e1fac4d2a41a2b8fbe9ae97f034bc7a7f" exitCode=0
Oct 31 14:03:21.134277 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:21.134091    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"
Oct 31 14:03:21.134277 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:21.134089    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj" event={"ID":"d5e6e17a-543e-4373-ad93-e0ed60c74334","Type":"ContainerDied","Data":"b2c4e91a017758f0511a49cac163f50e1fac4d2a41a2b8fbe9ae97f034bc7a7f"}
Oct 31 14:03:21.134277 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:21.134213    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj" event={"ID":"d5e6e17a-543e-4373-ad93-e0ed60c74334","Type":"ContainerDied","Data":"bf81813fa6df6a74c36fd39dde7f3ed59cda1af5fc71da5afb5ab9cd3ab775b7"}
Oct 31 14:03:21.134277 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:21.134237    4428 scope.go:122] "RemoveContainer" containerID="b2c4e91a017758f0511a49cac163f50e1fac4d2a41a2b8fbe9ae97f034bc7a7f"
Oct 31 14:03:21.145244 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:21.145214    4428 scope.go:122] "RemoveContainer" containerID="f4c42ef64539004b74cd50af897ac2cedec54f5db0d8bb70a148211da1ad9041"
Oct 31 14:03:21.155987 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:21.155786    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"]
Oct 31 14:03:21.156086 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:21.156040    4428 scope.go:122] "RemoveContainer" containerID="b2c4e91a017758f0511a49cac163f50e1fac4d2a41a2b8fbe9ae97f034bc7a7f"
Oct 31 14:03:21.156422 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:03:21.156403    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c4e91a017758f0511a49cac163f50e1fac4d2a41a2b8fbe9ae97f034bc7a7f\": container with ID starting with b2c4e91a017758f0511a49cac163f50e1fac4d2a41a2b8fbe9ae97f034bc7a7f not found: ID does not exist" containerID="b2c4e91a017758f0511a49cac163f50e1fac4d2a41a2b8fbe9ae97f034bc7a7f"
Oct 31 14:03:21.156472 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:21.156434    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c4e91a017758f0511a49cac163f50e1fac4d2a41a2b8fbe9ae97f034bc7a7f"} err="failed to get container status \"b2c4e91a017758f0511a49cac163f50e1fac4d2a41a2b8fbe9ae97f034bc7a7f\": rpc error: code = NotFound desc = could not find container \"b2c4e91a017758f0511a49cac163f50e1fac4d2a41a2b8fbe9ae97f034bc7a7f\": container with ID starting with b2c4e91a017758f0511a49cac163f50e1fac4d2a41a2b8fbe9ae97f034bc7a7f not found: ID does not exist"
Oct 31 14:03:21.156516 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:21.156474    4428 scope.go:122] "RemoveContainer" containerID="f4c42ef64539004b74cd50af897ac2cedec54f5db0d8bb70a148211da1ad9041"
Oct 31 14:03:21.156744 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:03:21.156724    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c42ef64539004b74cd50af897ac2cedec54f5db0d8bb70a148211da1ad9041\": container with ID starting with f4c42ef64539004b74cd50af897ac2cedec54f5db0d8bb70a148211da1ad9041 not found: ID does not exist" containerID="f4c42ef64539004b74cd50af897ac2cedec54f5db0d8bb70a148211da1ad9041"
Oct 31 14:03:21.156820 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:21.156754    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c42ef64539004b74cd50af897ac2cedec54f5db0d8bb70a148211da1ad9041"} err="failed to get container status \"f4c42ef64539004b74cd50af897ac2cedec54f5db0d8bb70a148211da1ad9041\": rpc error: code = NotFound desc = could not find container \"f4c42ef64539004b74cd50af897ac2cedec54f5db0d8bb70a148211da1ad9041\": container with ID starting with f4c42ef64539004b74cd50af897ac2cedec54f5db0d8bb70a148211da1ad9041 not found: ID does not exist"
Oct 31 14:03:21.160432 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:21.160405    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-zrmtj"]
Oct 31 14:03:21.985496 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:21.985467    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:22.756321 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:22.756286    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e6e17a-543e-4373-ad93-e0ed60c74334" path="/var/lib/kubelet/pods/d5e6e17a-543e-4373-ad93-e0ed60c74334/volumes"
Oct 31 14:03:22.903167 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:22.903134    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:23.706987 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:23.706949    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:24.083895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:24.083833    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:24.610445 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:24.610405    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:25.416526 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:25.416483    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:26.076468 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:26.076428    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"]
Oct 31 14:03:26.076732 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:26.076693    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv" podUID="58a646d3-fa24-4bde-a717-14d3d3ff4c4e" containerName="payload-pre-processing" containerID="cri-o://9c305ee29030099c2830b3aba351d9195d508734f4c98f591eda4ce6280feb7a" gracePeriod=30
Oct 31 14:03:26.095624 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:26.095592    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:26.251238 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:26.251202    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:26.361684 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:26.361651    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"
Oct 31 14:03:26.381010 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:26.380973    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"]
Oct 31 14:03:26.391454 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:26.391426    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1b75f9-xfgrd"]
Oct 31 14:03:26.420515 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:26.420482    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:26.503711 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:26.500437    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a646d3-fa24-4bde-a717-14d3d3ff4c4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a646d3-fa24-4bde-a717-14d3d3ff4c4e-plugins-config-volume\") pod \"58a646d3-fa24-4bde-a717-14d3d3ff4c4e\" (UID: \"58a646d3-fa24-4bde-a717-14d3d3ff4c4e\") "
Oct 31 14:03:26.503711 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:26.500561    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a646d3-fa24-4bde-a717-14d3d3ff4c4e-kube-api-access-qnr5n\" (UniqueName: \"kubernetes.io/projected/58a646d3-fa24-4bde-a717-14d3d3ff4c4e-kube-api-access-qnr5n\") pod \"58a646d3-fa24-4bde-a717-14d3d3ff4c4e\" (UID: \"58a646d3-fa24-4bde-a717-14d3d3ff4c4e\") "
Oct 31 14:03:26.503711 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:26.501208    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a646d3-fa24-4bde-a717-14d3d3ff4c4e-plugins-config-volume" pod "58a646d3-fa24-4bde-a717-14d3d3ff4c4e" (UID: "58a646d3-fa24-4bde-a717-14d3d3ff4c4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:03:26.504660 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:26.504613    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a646d3-fa24-4bde-a717-14d3d3ff4c4e-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:26.507939 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:26.507912    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a646d3-fa24-4bde-a717-14d3d3ff4c4e-kube-api-access-qnr5n" pod "58a646d3-fa24-4bde-a717-14d3d3ff4c4e" (UID: "58a646d3-fa24-4bde-a717-14d3d3ff4c4e"). InnerVolumeSpecName "kube-api-access-qnr5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:26.573749 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:26.573717    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:26.605859 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:26.605775    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnr5n\" (UniqueName: \"kubernetes.io/projected/58a646d3-fa24-4bde-a717-14d3d3ff4c4e-kube-api-access-qnr5n\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:26.754410 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:26.754378    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58d4555-ef4a-4a23-9438-8fad9bae554e" path="/var/lib/kubelet/pods/f58d4555-ef4a-4a23-9438-8fad9bae554e/volumes"
Oct 31 14:03:26.937927 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:26.937801    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:27.176539 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:27.176503    4428 generic.go:356] "Generic (PLEG): container finished" podID="58a646d3-fa24-4bde-a717-14d3d3ff4c4e" containerID="9c305ee29030099c2830b3aba351d9195d508734f4c98f591eda4ce6280feb7a" exitCode=0
Oct 31 14:03:27.176768 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:27.176550    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"
Oct 31 14:03:27.176768 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:27.176595    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv" event={"ID":"58a646d3-fa24-4bde-a717-14d3d3ff4c4e","Type":"ContainerDied","Data":"9c305ee29030099c2830b3aba351d9195d508734f4c98f591eda4ce6280feb7a"}
Oct 31 14:03:27.176768 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:27.176653    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv" event={"ID":"58a646d3-fa24-4bde-a717-14d3d3ff4c4e","Type":"ContainerDied","Data":"47aea849db024281c82328462b1065670f6d577e8ed54a7f429df9bb33583542"}
Oct 31 14:03:27.176768 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:27.176683    4428 scope.go:122] "RemoveContainer" containerID="9c305ee29030099c2830b3aba351d9195d508734f4c98f591eda4ce6280feb7a"
Oct 31 14:03:27.186794 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:27.186776    4428 scope.go:122] "RemoveContainer" containerID="9c305ee29030099c2830b3aba351d9195d508734f4c98f591eda4ce6280feb7a"
Oct 31 14:03:27.187114 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:03:27.187092    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c305ee29030099c2830b3aba351d9195d508734f4c98f591eda4ce6280feb7a\": container with ID starting with 9c305ee29030099c2830b3aba351d9195d508734f4c98f591eda4ce6280feb7a not found: ID does not exist" containerID="9c305ee29030099c2830b3aba351d9195d508734f4c98f591eda4ce6280feb7a"
Oct 31 14:03:27.187196 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:27.187126    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c305ee29030099c2830b3aba351d9195d508734f4c98f591eda4ce6280feb7a"} err="failed to get container status \"9c305ee29030099c2830b3aba351d9195d508734f4c98f591eda4ce6280feb7a\": rpc error: code = NotFound desc = could not find container \"9c305ee29030099c2830b3aba351d9195d508734f4c98f591eda4ce6280feb7a\": container with ID starting with 9c305ee29030099c2830b3aba351d9195d508734f4c98f591eda4ce6280feb7a not found: ID does not exist"
Oct 31 14:03:27.196879 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:27.196809    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"]
Oct 31 14:03:27.203946 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:27.203923    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1b75f9-646d8c996c-rscwv"]
Oct 31 14:03:27.416723 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:27.416627    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:28.207329 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:28.207294    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:28.755030 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:28.754994    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a646d3-fa24-4bde-a717-14d3d3ff4c4e" path="/var/lib/kubelet/pods/58a646d3-fa24-4bde-a717-14d3d3ff4c4e/volumes"
Oct 31 14:03:29.181912 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:29.181785    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:29.917956 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:29.917923    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:30.690499 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:30.690445    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:31.505338 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:31.505285    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:31.695761 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:31.695711    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:32.322226 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:32.322183    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:33.111642 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:33.111598    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:33.906306 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:33.906237    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:34.059195 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.059157    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:34.296949 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.296916    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:34.710956 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.710474    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:34.741305 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741261    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"]
Oct 31 14:03:34.741776 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741757    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d5e6e17a-543e-4373-ad93-e0ed60c74334" containerName="istio-proxy"
Oct 31 14:03:34.741837 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741780    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e6e17a-543e-4373-ad93-e0ed60c74334" containerName="istio-proxy"
Oct 31 14:03:34.741837 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741796    4428 cpu_manager.go:414] "containerMap: removing container" podUID="8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" containerName="istio-proxy"
Oct 31 14:03:34.741837 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741801    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" containerName="istio-proxy"
Oct 31 14:03:34.741837 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741817    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d5e6e17a-543e-4373-ad93-e0ed60c74334" containerName="istio-proxy"
Oct 31 14:03:34.741837 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741822    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e6e17a-543e-4373-ad93-e0ed60c74334" containerName="istio-proxy"
Oct 31 14:03:34.741837 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741829    4428 cpu_manager.go:414] "containerMap: removing container" podUID="f58d4555-ef4a-4a23-9438-8fad9bae554e" containerName="revoke-keys"
Oct 31 14:03:34.741837 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741834    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58d4555-ef4a-4a23-9438-8fad9bae554e" containerName="revoke-keys"
Oct 31 14:03:34.742147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741844    4428 cpu_manager.go:414] "containerMap: removing container" podUID="58a646d3-fa24-4bde-a717-14d3d3ff4c4e" containerName="payload-pre-processing"
Oct 31 14:03:34.742147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741849    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a646d3-fa24-4bde-a717-14d3d3ff4c4e" containerName="payload-pre-processing"
Oct 31 14:03:34.742147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741858    4428 cpu_manager.go:414] "containerMap: removing container" podUID="8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" containerName="istio-proxy"
Oct 31 14:03:34.742147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741878    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" containerName="istio-proxy"
Oct 31 14:03:34.742147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741965    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" containerName="istio-proxy"
Oct 31 14:03:34.742147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741978    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e6e17a-543e-4373-ad93-e0ed60c74334" containerName="istio-proxy"
Oct 31 14:03:34.742147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741984    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5a5785-d16f-4b1f-ae6d-5f6c0fa50c21" containerName="istio-proxy"
Oct 31 14:03:34.742147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741991    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58d4555-ef4a-4a23-9438-8fad9bae554e" containerName="revoke-keys"
Oct 31 14:03:34.742147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.741998    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a646d3-fa24-4bde-a717-14d3d3ff4c4e" containerName="payload-pre-processing"
Oct 31 14:03:34.745614 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.745593    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"
Oct 31 14:03:34.757288 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.757239    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"]
Oct 31 14:03:34.829526 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.829480    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:03:34.829739 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.829537    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:03:34.890521 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.890474    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt\" (UID: \"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"
Oct 31 14:03:34.890749 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.890542    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6csq\" (UniqueName: \"kubernetes.io/projected/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-kube-api-access-r6csq\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt\" (UID: \"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"
Oct 31 14:03:34.890860 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.890834    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt\" (UID: \"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"
Oct 31 14:03:34.992487 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.992386    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt\" (UID: \"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"
Oct 31 14:03:34.992487 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.992430    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6csq\" (UniqueName: \"kubernetes.io/projected/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-kube-api-access-r6csq\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt\" (UID: \"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"
Oct 31 14:03:34.992769 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.992575    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt\" (UID: \"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"
Oct 31 14:03:34.993202 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.993181    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt\" (UID: \"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"
Oct 31 14:03:34.995060 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:34.995041    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt\" (UID: \"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"
Oct 31 14:03:35.005699 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:35.005666    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6csq\" (UniqueName: \"kubernetes.io/projected/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-kube-api-access-r6csq\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt\" (UID: \"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"
Oct 31 14:03:35.062558 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:35.062522    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"
Oct 31 14:03:35.345243 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:35.345207    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"]
Oct 31 14:03:35.347589 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:03:35.347559    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37ae6e0_4b21_4d4d_b8c9_c49a68ebec2f.slice/crio-a581f391f826a00aafddb810c0fcfaebbce597ad8c18fbf4bc8145fede2c9aba WatchSource:0}: Error finding container a581f391f826a00aafddb810c0fcfaebbce597ad8c18fbf4bc8145fede2c9aba: Status 404 returned error can't find the container with id a581f391f826a00aafddb810c0fcfaebbce597ad8c18fbf4bc8145fede2c9aba
Oct 31 14:03:35.697641 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:35.697602    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:36.236771 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:36.236724    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt" event={"ID":"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f","Type":"ContainerStarted","Data":"6cd47b9fa9538d288cc1124d4e141380a2e2ca62f0bc521262354f83dabada5d"}
Oct 31 14:03:36.237010 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:36.236793    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt" event={"ID":"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f","Type":"ContainerStarted","Data":"a581f391f826a00aafddb810c0fcfaebbce597ad8c18fbf4bc8145fede2c9aba"}
Oct 31 14:03:36.256989 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:36.256911    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt" podStartSLOduration=2.256861101 podStartE2EDuration="2.256861101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:03:36.256026207 +0000 UTC m=+3613.654374832" watchObservedRunningTime="2026-10-31 14:03:36.256861101 +0000 UTC m=+3613.655209726"
Oct 31 14:03:36.487038 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:36.487000    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:37.243921 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:37.243848    4428 generic.go:356] "Generic (PLEG): container finished" podID="b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f" containerID="6cd47b9fa9538d288cc1124d4e141380a2e2ca62f0bc521262354f83dabada5d" exitCode=0
Oct 31 14:03:37.243921 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:37.243910    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt" event={"ID":"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f","Type":"ContainerDied","Data":"6cd47b9fa9538d288cc1124d4e141380a2e2ca62f0bc521262354f83dabada5d"}
Oct 31 14:03:37.383046 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:37.383007    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:38.139776 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:38.139737    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:38.395658 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:38.395588    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"
Oct 31 14:03:38.437782 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:38.437714    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-maas-api-openshift-ca\") pod \"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f\" (UID: \"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f\") "
Oct 31 14:03:38.438106 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:38.438032    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-maas-api-xks-ca\") pod \"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f\" (UID: \"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f\") "
Oct 31 14:03:38.438106 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:38.438077    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-kube-api-access-r6csq\" (UniqueName: \"kubernetes.io/projected/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-kube-api-access-r6csq\") pod \"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f\" (UID: \"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f\") "
Oct 31 14:03:38.438664 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:38.438317    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-maas-api-openshift-ca" pod "b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f" (UID: "b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:03:38.438664 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:38.438638    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:38.440905 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:38.440835    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-maas-api-xks-ca" pod "b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f" (UID: "b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:03:38.441116 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:38.441090    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-kube-api-access-r6csq" pod "b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f" (UID: "b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f"). InnerVolumeSpecName "kube-api-access-r6csq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:38.539627 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:38.539596    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:38.539627 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:38.539624    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6csq\" (UniqueName: \"kubernetes.io/projected/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f-kube-api-access-r6csq\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:39.055314 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:39.055222    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:39.257313 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:39.257277    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt" event={"ID":"b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f","Type":"ContainerDied","Data":"a581f391f826a00aafddb810c0fcfaebbce597ad8c18fbf4bc8145fede2c9aba"}
Oct 31 14:03:39.257313 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:39.257312    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"
Oct 31 14:03:39.257775 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:39.257331    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a581f391f826a00aafddb810c0fcfaebbce597ad8c18fbf4bc8145fede2c9aba"
Oct 31 14:03:39.708996 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:39.708962    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:40.596110 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:40.596069    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:41.412232 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:41.412195    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:42.289994 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:42.289960    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:43.110156 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:43.110121    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:43.900059 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:43.900019    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:44.745699 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:44.745668    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:44.808799 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:44.808759    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"]
Oct 31 14:03:44.972030 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:44.971995    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:45.158587 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:45.158427    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:45.597289 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:45.597258    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:46.219303 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:46.219275    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:46.392914 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:46.392881    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:46.448786 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:46.448747    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"]
Oct 31 14:03:46.452024 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:46.451994    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a39e7e-shxgt"]
Oct 31 14:03:46.547563 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:46.547462    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:46.712506 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:46.712456    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:46.754634 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:46.754603    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f" path="/var/lib/kubelet/pods/b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f/volumes"
Oct 31 14:03:47.002372 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:47.002340    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:47.425320 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:47.425239    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:48.310766 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:48.310722    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:48.741015 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:03:48.740977    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:03:49.125609 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:49.125519    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:49.809999 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:49.809959    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726" podUID="2d678e30-767d-48e8-8c7a-74c31e08ed30" containerName="payload-pre-processing" containerID="cri-o://2fc06fc19f62370a16d79829d3d20278875294a17f6059de69c29bf6eb599181" gracePeriod=25
Oct 31 14:03:50.019399 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:50.019348    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:50.798356 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:50.798301    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:51.501923 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:51.501887    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:51.657357 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:51.657319    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:51.813207 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:51.813109    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:52.512308 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:52.512277    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:53.405806 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:53.405759    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:54.033666 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:54.033634    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:54.189298 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:54.189259    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:54.287690 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:54.287595    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"]
Oct 31 14:03:54.287912 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:54.287884    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh" podUID="5701fca5-38a3-4a28-86b3-55124cb6f54e" containerName="istio-proxy" containerID="cri-o://c364208074403e6d29da90aa9d609acadada0a9da75616be12046e3d111f493c" gracePeriod=30
Oct 31 14:03:54.356715 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:54.356681    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:54.595218 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:54.595117    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:54.890577 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:54.890483    4428 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:03:54.890577 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:54.890562    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh" podUID="5701fca5-38a3-4a28-86b3-55124cb6f54e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:03:55.356967 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:55.356932    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:55.371580 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:55.371541    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726" podUID="2d678e30-767d-48e8-8c7a-74c31e08ed30" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:03:56.276315 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:56.276280    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:56.981758 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:56.981719    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:57.802551 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:57.802514    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:58.712613 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:58.712580    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:59.551287 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.551259    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:03:59.563971 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.563900    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 14:03:59.651080 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651046    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-podinfo\") pod \"5701fca5-38a3-4a28-86b3-55124cb6f54e\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") "
Oct 31 14:03:59.651396 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651091    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-data\") pod \"5701fca5-38a3-4a28-86b3-55124cb6f54e\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") "
Oct 31 14:03:59.651396 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651116    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-credential-socket\") pod \"5701fca5-38a3-4a28-86b3-55124cb6f54e\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") "
Oct 31 14:03:59.651396 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651161    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-workload-certs\") pod \"5701fca5-38a3-4a28-86b3-55124cb6f54e\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") "
Oct 31 14:03:59.651396 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651230    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-envoy\") pod \"5701fca5-38a3-4a28-86b3-55124cb6f54e\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") "
Oct 31 14:03:59.651396 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651263    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-workload-socket\") pod \"5701fca5-38a3-4a28-86b3-55124cb6f54e\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") "
Oct 31 14:03:59.651396 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651324    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5701fca5-38a3-4a28-86b3-55124cb6f54e-kube-api-access-cnlbd\" (UniqueName: \"kubernetes.io/projected/5701fca5-38a3-4a28-86b3-55124cb6f54e-kube-api-access-cnlbd\") pod \"5701fca5-38a3-4a28-86b3-55124cb6f54e\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") "
Oct 31 14:03:59.651396 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651358    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-token\" (UniqueName: \"kubernetes.io/projected/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-token\") pod \"5701fca5-38a3-4a28-86b3-55124cb6f54e\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") "
Oct 31 14:03:59.651834 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651421    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5701fca5-38a3-4a28-86b3-55124cb6f54e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5701fca5-38a3-4a28-86b3-55124cb6f54e-istiod-ca-cert\") pod \"5701fca5-38a3-4a28-86b3-55124cb6f54e\" (UID: \"5701fca5-38a3-4a28-86b3-55124cb6f54e\") "
Oct 31 14:03:59.651834 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651456    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-credential-socket" pod "5701fca5-38a3-4a28-86b3-55124cb6f54e" (UID: "5701fca5-38a3-4a28-86b3-55124cb6f54e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:59.651834 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651498    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-data" pod "5701fca5-38a3-4a28-86b3-55124cb6f54e" (UID: "5701fca5-38a3-4a28-86b3-55124cb6f54e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:59.651834 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651544    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-workload-certs" pod "5701fca5-38a3-4a28-86b3-55124cb6f54e" (UID: "5701fca5-38a3-4a28-86b3-55124cb6f54e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:59.651834 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651602    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-workload-socket" pod "5701fca5-38a3-4a28-86b3-55124cb6f54e" (UID: "5701fca5-38a3-4a28-86b3-55124cb6f54e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:59.652359 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651862    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:59.652359 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651908    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5701fca5-38a3-4a28-86b3-55124cb6f54e-istiod-ca-cert" pod "5701fca5-38a3-4a28-86b3-55124cb6f54e" (UID: "5701fca5-38a3-4a28-86b3-55124cb6f54e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:03:59.652359 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651922    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:59.652359 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651937    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:59.652359 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.651950    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:59.653512 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.653488    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-podinfo" pod "5701fca5-38a3-4a28-86b3-55124cb6f54e" (UID: "5701fca5-38a3-4a28-86b3-55124cb6f54e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:03:59.653717 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.653696    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-envoy" pod "5701fca5-38a3-4a28-86b3-55124cb6f54e" (UID: "5701fca5-38a3-4a28-86b3-55124cb6f54e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:03:59.653888 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.653852    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-token" pod "5701fca5-38a3-4a28-86b3-55124cb6f54e" (UID: "5701fca5-38a3-4a28-86b3-55124cb6f54e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:59.654048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.654025    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5701fca5-38a3-4a28-86b3-55124cb6f54e-kube-api-access-cnlbd" pod "5701fca5-38a3-4a28-86b3-55124cb6f54e" (UID: "5701fca5-38a3-4a28-86b3-55124cb6f54e"). InnerVolumeSpecName "kube-api-access-cnlbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:03:59.752565 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.752530    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:59.752565 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.752564    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnlbd\" (UniqueName: \"kubernetes.io/projected/5701fca5-38a3-4a28-86b3-55124cb6f54e-kube-api-access-cnlbd\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:59.753082 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.752576    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:59.753082 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.752585    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5701fca5-38a3-4a28-86b3-55124cb6f54e-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:03:59.753082 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:03:59.752593    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5701fca5-38a3-4a28-86b3-55124cb6f54e-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:00.399140 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:00.399035    4428 generic.go:356] "Generic (PLEG): container finished" podID="5701fca5-38a3-4a28-86b3-55124cb6f54e" containerID="c364208074403e6d29da90aa9d609acadada0a9da75616be12046e3d111f493c" exitCode=0
Oct 31 14:04:00.399140 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:00.399109    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"
Oct 31 14:04:00.399487 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:00.399099    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh" event={"ID":"5701fca5-38a3-4a28-86b3-55124cb6f54e","Type":"ContainerDied","Data":"c364208074403e6d29da90aa9d609acadada0a9da75616be12046e3d111f493c"}
Oct 31 14:04:00.399487 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:00.399273    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh" event={"ID":"5701fca5-38a3-4a28-86b3-55124cb6f54e","Type":"ContainerDied","Data":"8f0c38cefc18d6936ee1c6f80ad8c028fc03c4c6f2338f59dd2fa76de783110e"}
Oct 31 14:04:00.399487 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:00.399314    4428 scope.go:122] "RemoveContainer" containerID="c364208074403e6d29da90aa9d609acadada0a9da75616be12046e3d111f493c"
Oct 31 14:04:00.406440 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:00.406417    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:00.415957 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:00.415938    4428 scope.go:122] "RemoveContainer" containerID="efe28536f4ea94d65dbf1603e4354ba9c1982b453fc165400e6ca54cec638912"
Oct 31 14:04:00.425909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:00.425890    4428 scope.go:122] "RemoveContainer" containerID="c364208074403e6d29da90aa9d609acadada0a9da75616be12046e3d111f493c"
Oct 31 14:04:00.426261 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:04:00.426239    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c364208074403e6d29da90aa9d609acadada0a9da75616be12046e3d111f493c\": container with ID starting with c364208074403e6d29da90aa9d609acadada0a9da75616be12046e3d111f493c not found: ID does not exist" containerID="c364208074403e6d29da90aa9d609acadada0a9da75616be12046e3d111f493c"
Oct 31 14:04:00.426323 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:00.426271    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c364208074403e6d29da90aa9d609acadada0a9da75616be12046e3d111f493c"} err="failed to get container status \"c364208074403e6d29da90aa9d609acadada0a9da75616be12046e3d111f493c\": rpc error: code = NotFound desc = could not find container \"c364208074403e6d29da90aa9d609acadada0a9da75616be12046e3d111f493c\": container with ID starting with c364208074403e6d29da90aa9d609acadada0a9da75616be12046e3d111f493c not found: ID does not exist"
Oct 31 14:04:00.426323 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:00.426289    4428 scope.go:122] "RemoveContainer" containerID="efe28536f4ea94d65dbf1603e4354ba9c1982b453fc165400e6ca54cec638912"
Oct 31 14:04:00.426518 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:04:00.426502    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe28536f4ea94d65dbf1603e4354ba9c1982b453fc165400e6ca54cec638912\": container with ID starting with efe28536f4ea94d65dbf1603e4354ba9c1982b453fc165400e6ca54cec638912 not found: ID does not exist" containerID="efe28536f4ea94d65dbf1603e4354ba9c1982b453fc165400e6ca54cec638912"
Oct 31 14:04:00.426568 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:00.426522    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe28536f4ea94d65dbf1603e4354ba9c1982b453fc165400e6ca54cec638912"} err="failed to get container status \"efe28536f4ea94d65dbf1603e4354ba9c1982b453fc165400e6ca54cec638912\": rpc error: code = NotFound desc = could not find container \"efe28536f4ea94d65dbf1603e4354ba9c1982b453fc165400e6ca54cec638912\": container with ID starting with efe28536f4ea94d65dbf1603e4354ba9c1982b453fc165400e6ca54cec638912 not found: ID does not exist"
Oct 31 14:04:00.431737 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:00.431708    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"]
Oct 31 14:04:00.434846 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:00.434824    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-svkjh"]
Oct 31 14:04:00.754773 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:00.754738    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5701fca5-38a3-4a28-86b3-55124cb6f54e" path="/var/lib/kubelet/pods/5701fca5-38a3-4a28-86b3-55124cb6f54e/volumes"
Oct 31 14:04:01.193104 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:01.193011    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:02.002354 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:02.002321    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:02.808045 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:02.808003    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:03.706065 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:03.706013    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:04.415322 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:04.415233    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:04.833110 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:04.833072    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"]
Oct 31 14:04:04.845794 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:04.845763    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"]
Oct 31 14:04:04.962569 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:04.962535    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:05.121086 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:05.120998    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:05.321162 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:05.321126    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:05.371380 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:05.371295    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726" podUID="2d678e30-767d-48e8-8c7a-74c31e08ed30" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:04:06.005319 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:06.005287    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:06.708756 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:06.708723    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:06.898710 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:06.898672    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:07.051903 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:07.051815    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:07.252301 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:07.252268    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:07.521372 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:07.521326    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:07.927343 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:07.927222    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:08.702802 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:08.702758    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:09.532846 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:09.532801    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:09.833847 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:09.833813    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh" podUID="81d7c2b0-44d4-4b9f-8000-aaf84a56c722" containerName="payload-processing" containerID="cri-o://b28b7442c1fc415aecd36ad7a5644853443e4be61d21f874b31fd618ab5997c4" gracePeriod=25
Oct 31 14:04:09.847072 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:09.847039    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs" podUID="bc620c07-7229-4b69-9686-6ae7786b869f" containerName="payload-pre-processing" containerID="cri-o://c79578ee5cdd7307fb214c52486ca078f36f4b2e6fad0f846d74021b2edc5378" gracePeriod=25
Oct 31 14:04:10.066828 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.066765    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"
Oct 31 14:04:10.160853 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.160817    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d678e30-767d-48e8-8c7a-74c31e08ed30-kube-api-access-cbrd6\" (UniqueName: \"kubernetes.io/projected/2d678e30-767d-48e8-8c7a-74c31e08ed30-kube-api-access-cbrd6\") pod \"2d678e30-767d-48e8-8c7a-74c31e08ed30\" (UID: \"2d678e30-767d-48e8-8c7a-74c31e08ed30\") "
Oct 31 14:04:10.161070 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.160943    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d678e30-767d-48e8-8c7a-74c31e08ed30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d678e30-767d-48e8-8c7a-74c31e08ed30-plugins-config-volume\") pod \"2d678e30-767d-48e8-8c7a-74c31e08ed30\" (UID: \"2d678e30-767d-48e8-8c7a-74c31e08ed30\") "
Oct 31 14:04:10.161370 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.161346    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d678e30-767d-48e8-8c7a-74c31e08ed30-plugins-config-volume" pod "2d678e30-767d-48e8-8c7a-74c31e08ed30" (UID: "2d678e30-767d-48e8-8c7a-74c31e08ed30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:04:10.163097 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.163059    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d678e30-767d-48e8-8c7a-74c31e08ed30-kube-api-access-cbrd6" pod "2d678e30-767d-48e8-8c7a-74c31e08ed30" (UID: "2d678e30-767d-48e8-8c7a-74c31e08ed30"). InnerVolumeSpecName "kube-api-access-cbrd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:04:10.261721 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.261678    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d678e30-767d-48e8-8c7a-74c31e08ed30-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:10.261721 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.261712    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbrd6\" (UniqueName: \"kubernetes.io/projected/2d678e30-767d-48e8-8c7a-74c31e08ed30-kube-api-access-cbrd6\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:10.406305 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.406203    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:10.459969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.459931    4428 generic.go:356] "Generic (PLEG): container finished" podID="2d678e30-767d-48e8-8c7a-74c31e08ed30" containerID="2fc06fc19f62370a16d79829d3d20278875294a17f6059de69c29bf6eb599181" exitCode=0
Oct 31 14:04:10.460180 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.460008    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"
Oct 31 14:04:10.460180 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.460033    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726" event={"ID":"2d678e30-767d-48e8-8c7a-74c31e08ed30","Type":"ContainerDied","Data":"2fc06fc19f62370a16d79829d3d20278875294a17f6059de69c29bf6eb599181"}
Oct 31 14:04:10.460180 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.460127    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726" event={"ID":"2d678e30-767d-48e8-8c7a-74c31e08ed30","Type":"ContainerDied","Data":"c21ef0776925caa5f952e9efcabb219ef732221d4e368a4a30e072d8ddf64796"}
Oct 31 14:04:10.460180 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.460152    4428 scope.go:122] "RemoveContainer" containerID="2fc06fc19f62370a16d79829d3d20278875294a17f6059de69c29bf6eb599181"
Oct 31 14:04:10.470566 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.470543    4428 scope.go:122] "RemoveContainer" containerID="2fc06fc19f62370a16d79829d3d20278875294a17f6059de69c29bf6eb599181"
Oct 31 14:04:10.470860 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:04:10.470841    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc06fc19f62370a16d79829d3d20278875294a17f6059de69c29bf6eb599181\": container with ID starting with 2fc06fc19f62370a16d79829d3d20278875294a17f6059de69c29bf6eb599181 not found: ID does not exist" containerID="2fc06fc19f62370a16d79829d3d20278875294a17f6059de69c29bf6eb599181"
Oct 31 14:04:10.470950 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.470898    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc06fc19f62370a16d79829d3d20278875294a17f6059de69c29bf6eb599181"} err="failed to get container status \"2fc06fc19f62370a16d79829d3d20278875294a17f6059de69c29bf6eb599181\": rpc error: code = NotFound desc = could not find container \"2fc06fc19f62370a16d79829d3d20278875294a17f6059de69c29bf6eb599181\": container with ID starting with 2fc06fc19f62370a16d79829d3d20278875294a17f6059de69c29bf6eb599181 not found: ID does not exist"
Oct 31 14:04:10.488269 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.488238    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"]
Oct 31 14:04:10.494236 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.494211    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a39e7e-547df6896c-n9726"]
Oct 31 14:04:10.754062 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:10.754019    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d678e30-767d-48e8-8c7a-74c31e08ed30" path="/var/lib/kubelet/pods/2d678e30-767d-48e8-8c7a-74c31e08ed30/volumes"
Oct 31 14:04:11.197839 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:11.197717    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:12.072240 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:12.072196    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:12.826290 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:12.826255    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:13.607380 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:13.607328    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:14.479183 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:14.479143    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh" podUID="81d7c2b0-44d4-4b9f-8000-aaf84a56c722" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:04:14.479606 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:14.479144    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs" podUID="bc620c07-7229-4b69-9686-6ae7786b869f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:04:14.608249 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:14.608213    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:15.209895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:15.209821    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:15.419250 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:15.419218    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:15.459737 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:15.459705    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"]
Oct 31 14:04:15.460039 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:15.459949    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z" podUID="4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" containerName="istio-proxy" containerID="cri-o://e43e45a6926c728257d09f69f9956393b73b4d8cbde1e8db00c064a770633fce" gracePeriod=30
Oct 31 14:04:15.590189 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:15.590146    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:15.822671 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:15.822559    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:16.032077 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:16.032042    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:16.215744 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:16.215710    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:16.544745 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:16.544660    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:17.284558 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:17.284521    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:18.146720 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:18.146690    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:18.741406 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:04:18.741361    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:04:19.124330 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:19.124235    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:19.904417 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:19.904387    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:20.723394 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.723368    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 14:04:20.885828 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.885737    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-workload-socket\") pod \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") "
Oct 31 14:04:20.885828 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.885780    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-podinfo\") pod \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") "
Oct 31 14:04:20.885828 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.885827    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-token\" (UniqueName: \"kubernetes.io/projected/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-token\") pod \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") "
Oct 31 14:04:20.886222 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.885854    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istiod-ca-cert\") pod \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") "
Oct 31 14:04:20.886222 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.885931    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-workload-certs\") pod \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") "
Oct 31 14:04:20.886222 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.885962    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-credential-socket\") pod \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") "
Oct 31 14:04:20.886222 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.885989    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-data\") pod \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") "
Oct 31 14:04:20.886222 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.886025    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-kube-api-access-8sm6s\" (UniqueName: \"kubernetes.io/projected/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-kube-api-access-8sm6s\") pod \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") "
Oct 31 14:04:20.886222 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.886046    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-workload-socket" pod "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" (UID: "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:20.886222 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.886064    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-envoy\") pod \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\" (UID: \"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af\") "
Oct 31 14:04:20.886793 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.886238    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-workload-certs" pod "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" (UID: "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:20.886793 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.886271    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-credential-socket" pod "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" (UID: "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:20.886793 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.886281    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-data" pod "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" (UID: "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:20.886793 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.886337    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istiod-ca-cert" pod "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" (UID: "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:04:20.886793 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.886532    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:20.886793 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.886550    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:20.886793 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.886560    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:20.886793 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.886586    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:20.886793 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.886639    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:20.888348 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.888325    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-podinfo" pod "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" (UID: "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:04:20.888506 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.888491    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-envoy" pod "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" (UID: "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:20.888948 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.888921    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-kube-api-access-8sm6s" pod "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" (UID: "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af"). InnerVolumeSpecName "kube-api-access-8sm6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:04:20.889059 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.888946    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-token" pod "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" (UID: "4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:04:20.905289 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.905254    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:20.987316 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.987278    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:20.987525 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.987345    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:20.987525 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.987359    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sm6s\" (UniqueName: \"kubernetes.io/projected/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-kube-api-access-8sm6s\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:20.987525 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:20.987375    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:21.099642 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:21.099604    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:21.534157 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:21.534118    4428 generic.go:356] "Generic (PLEG): container finished" podID="4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" containerID="e43e45a6926c728257d09f69f9956393b73b4d8cbde1e8db00c064a770633fce" exitCode=0
Oct 31 14:04:21.534389 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:21.534185    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"
Oct 31 14:04:21.534389 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:21.534181    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z" event={"ID":"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af","Type":"ContainerDied","Data":"e43e45a6926c728257d09f69f9956393b73b4d8cbde1e8db00c064a770633fce"}
Oct 31 14:04:21.534389 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:21.534276    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z" event={"ID":"4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af","Type":"ContainerDied","Data":"e7027e67f2e35e1f4e74e1fe0e3d53b75e9d1e9f8bf6e6ad2e829ca9e5575808"}
Oct 31 14:04:21.534389 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:21.534306    4428 scope.go:122] "RemoveContainer" containerID="e43e45a6926c728257d09f69f9956393b73b4d8cbde1e8db00c064a770633fce"
Oct 31 14:04:21.546027 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:21.546008    4428 scope.go:122] "RemoveContainer" containerID="e43e45a6926c728257d09f69f9956393b73b4d8cbde1e8db00c064a770633fce"
Oct 31 14:04:21.546341 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:04:21.546323    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e43e45a6926c728257d09f69f9956393b73b4d8cbde1e8db00c064a770633fce\": container with ID starting with e43e45a6926c728257d09f69f9956393b73b4d8cbde1e8db00c064a770633fce not found: ID does not exist" containerID="e43e45a6926c728257d09f69f9956393b73b4d8cbde1e8db00c064a770633fce"
Oct 31 14:04:21.546419 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:21.546350    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e43e45a6926c728257d09f69f9956393b73b4d8cbde1e8db00c064a770633fce"} err="failed to get container status \"e43e45a6926c728257d09f69f9956393b73b4d8cbde1e8db00c064a770633fce\": rpc error: code = NotFound desc = could not find container \"e43e45a6926c728257d09f69f9956393b73b4d8cbde1e8db00c064a770633fce\": container with ID starting with e43e45a6926c728257d09f69f9956393b73b4d8cbde1e8db00c064a770633fce not found: ID does not exist"
Oct 31 14:04:21.559242 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:21.559211    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"]
Oct 31 14:04:21.564334 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:21.564306    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-wcr5z"]
Oct 31 14:04:21.713915 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:21.713880    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:22.610152 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:22.610103    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:22.754062 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:22.754027    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" path="/var/lib/kubelet/pods/4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af/volumes"
Oct 31 14:04:23.396341 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:23.396310    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:24.188204 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:24.188167    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:24.478638 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:24.478598    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh" podUID="81d7c2b0-44d4-4b9f-8000-aaf84a56c722" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:04:24.478638 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:24.478611    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs" podUID="bc620c07-7229-4b69-9686-6ae7786b869f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:04:24.903346 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:24.903263    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:25.716136 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:25.716093    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:26.185973 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:26.185878    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"]
Oct 31 14:04:26.326631 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:26.326601    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:26.482850 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:26.482812    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:30.123606 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.123578    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"
Oct 31 14:04:30.156324 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.156296    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"
Oct 31 14:04:30.181088 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.181054    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc620c07-7229-4b69-9686-6ae7786b869f-kube-api-access-sjnnw\" (UniqueName: \"kubernetes.io/projected/bc620c07-7229-4b69-9686-6ae7786b869f-kube-api-access-sjnnw\") pod \"bc620c07-7229-4b69-9686-6ae7786b869f\" (UID: \"bc620c07-7229-4b69-9686-6ae7786b869f\") "
Oct 31 14:04:30.181313 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.181135    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d7c2b0-44d4-4b9f-8000-aaf84a56c722-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d7c2b0-44d4-4b9f-8000-aaf84a56c722-plugins-config-volume\") pod \"81d7c2b0-44d4-4b9f-8000-aaf84a56c722\" (UID: \"81d7c2b0-44d4-4b9f-8000-aaf84a56c722\") "
Oct 31 14:04:30.181313 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.181175    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc620c07-7229-4b69-9686-6ae7786b869f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc620c07-7229-4b69-9686-6ae7786b869f-plugins-config-volume\") pod \"bc620c07-7229-4b69-9686-6ae7786b869f\" (UID: \"bc620c07-7229-4b69-9686-6ae7786b869f\") "
Oct 31 14:04:30.181313 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.181219    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d7c2b0-44d4-4b9f-8000-aaf84a56c722-kube-api-access-7jzdp\" (UniqueName: \"kubernetes.io/projected/81d7c2b0-44d4-4b9f-8000-aaf84a56c722-kube-api-access-7jzdp\") pod \"81d7c2b0-44d4-4b9f-8000-aaf84a56c722\" (UID: \"81d7c2b0-44d4-4b9f-8000-aaf84a56c722\") "
Oct 31 14:04:30.181555 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.181519    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d7c2b0-44d4-4b9f-8000-aaf84a56c722-plugins-config-volume" pod "81d7c2b0-44d4-4b9f-8000-aaf84a56c722" (UID: "81d7c2b0-44d4-4b9f-8000-aaf84a56c722"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:04:30.182188 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.182163    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc620c07-7229-4b69-9686-6ae7786b869f-plugins-config-volume" pod "bc620c07-7229-4b69-9686-6ae7786b869f" (UID: "bc620c07-7229-4b69-9686-6ae7786b869f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:04:30.183815 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.183792    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc620c07-7229-4b69-9686-6ae7786b869f-kube-api-access-sjnnw" pod "bc620c07-7229-4b69-9686-6ae7786b869f" (UID: "bc620c07-7229-4b69-9686-6ae7786b869f"). InnerVolumeSpecName "kube-api-access-sjnnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:04:30.184170 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.184142    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d7c2b0-44d4-4b9f-8000-aaf84a56c722-kube-api-access-7jzdp" pod "81d7c2b0-44d4-4b9f-8000-aaf84a56c722" (UID: "81d7c2b0-44d4-4b9f-8000-aaf84a56c722"). InnerVolumeSpecName "kube-api-access-7jzdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:04:30.282320 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.282278    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jzdp\" (UniqueName: \"kubernetes.io/projected/81d7c2b0-44d4-4b9f-8000-aaf84a56c722-kube-api-access-7jzdp\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:30.282320 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.282311    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjnnw\" (UniqueName: \"kubernetes.io/projected/bc620c07-7229-4b69-9686-6ae7786b869f-kube-api-access-sjnnw\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:30.282320 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.282320    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d7c2b0-44d4-4b9f-8000-aaf84a56c722-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:30.282320 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.282328    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc620c07-7229-4b69-9686-6ae7786b869f-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:30.590119 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.590076    4428 generic.go:356] "Generic (PLEG): container finished" podID="bc620c07-7229-4b69-9686-6ae7786b869f" containerID="c79578ee5cdd7307fb214c52486ca078f36f4b2e6fad0f846d74021b2edc5378" exitCode=0
Oct 31 14:04:30.590377 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.590138    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"
Oct 31 14:04:30.590377 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.590160    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs" event={"ID":"bc620c07-7229-4b69-9686-6ae7786b869f","Type":"ContainerDied","Data":"c79578ee5cdd7307fb214c52486ca078f36f4b2e6fad0f846d74021b2edc5378"}
Oct 31 14:04:30.590377 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.590246    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs" event={"ID":"bc620c07-7229-4b69-9686-6ae7786b869f","Type":"ContainerDied","Data":"e3a754c80634416bd1584d13a954f21b5b4a69d24058f3c3f02f0d08b4533033"}
Oct 31 14:04:30.590377 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.590278    4428 scope.go:122] "RemoveContainer" containerID="c79578ee5cdd7307fb214c52486ca078f36f4b2e6fad0f846d74021b2edc5378"
Oct 31 14:04:30.591611 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.591584    4428 generic.go:356] "Generic (PLEG): container finished" podID="81d7c2b0-44d4-4b9f-8000-aaf84a56c722" containerID="b28b7442c1fc415aecd36ad7a5644853443e4be61d21f874b31fd618ab5997c4" exitCode=0
Oct 31 14:04:30.591727 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.591644    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"
Oct 31 14:04:30.591790 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.591727    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh" event={"ID":"81d7c2b0-44d4-4b9f-8000-aaf84a56c722","Type":"ContainerDied","Data":"b28b7442c1fc415aecd36ad7a5644853443e4be61d21f874b31fd618ab5997c4"}
Oct 31 14:04:30.591790 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.591781    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh" event={"ID":"81d7c2b0-44d4-4b9f-8000-aaf84a56c722","Type":"ContainerDied","Data":"92edce4e11f4266ab40faa3100e1470d340b8b70c216331adf7d85560cb91b59"}
Oct 31 14:04:30.602573 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.602554    4428 scope.go:122] "RemoveContainer" containerID="c79578ee5cdd7307fb214c52486ca078f36f4b2e6fad0f846d74021b2edc5378"
Oct 31 14:04:30.602897 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:04:30.602849    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79578ee5cdd7307fb214c52486ca078f36f4b2e6fad0f846d74021b2edc5378\": container with ID starting with c79578ee5cdd7307fb214c52486ca078f36f4b2e6fad0f846d74021b2edc5378 not found: ID does not exist" containerID="c79578ee5cdd7307fb214c52486ca078f36f4b2e6fad0f846d74021b2edc5378"
Oct 31 14:04:30.602969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.602918    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79578ee5cdd7307fb214c52486ca078f36f4b2e6fad0f846d74021b2edc5378"} err="failed to get container status \"c79578ee5cdd7307fb214c52486ca078f36f4b2e6fad0f846d74021b2edc5378\": rpc error: code = NotFound desc = could not find container \"c79578ee5cdd7307fb214c52486ca078f36f4b2e6fad0f846d74021b2edc5378\": container with ID starting with c79578ee5cdd7307fb214c52486ca078f36f4b2e6fad0f846d74021b2edc5378 not found: ID does not exist"
Oct 31 14:04:30.602969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.602935    4428 scope.go:122] "RemoveContainer" containerID="b28b7442c1fc415aecd36ad7a5644853443e4be61d21f874b31fd618ab5997c4"
Oct 31 14:04:30.612082 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.612061    4428 scope.go:122] "RemoveContainer" containerID="b28b7442c1fc415aecd36ad7a5644853443e4be61d21f874b31fd618ab5997c4"
Oct 31 14:04:30.612377 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:04:30.612355    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28b7442c1fc415aecd36ad7a5644853443e4be61d21f874b31fd618ab5997c4\": container with ID starting with b28b7442c1fc415aecd36ad7a5644853443e4be61d21f874b31fd618ab5997c4 not found: ID does not exist" containerID="b28b7442c1fc415aecd36ad7a5644853443e4be61d21f874b31fd618ab5997c4"
Oct 31 14:04:30.612461 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.612384    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28b7442c1fc415aecd36ad7a5644853443e4be61d21f874b31fd618ab5997c4"} err="failed to get container status \"b28b7442c1fc415aecd36ad7a5644853443e4be61d21f874b31fd618ab5997c4\": rpc error: code = NotFound desc = could not find container \"b28b7442c1fc415aecd36ad7a5644853443e4be61d21f874b31fd618ab5997c4\": container with ID starting with b28b7442c1fc415aecd36ad7a5644853443e4be61d21f874b31fd618ab5997c4 not found: ID does not exist"
Oct 31 14:04:30.616590 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.616566    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"]
Oct 31 14:04:30.624084 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.624063    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35d13e-76bf444c-bsqvs"]
Oct 31 14:04:30.637100 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.637073    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"]
Oct 31 14:04:30.641510 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.641486    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-35d13e-5857c666d9-fs4kh"]
Oct 31 14:04:30.755530 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.755497    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d7c2b0-44d4-4b9f-8000-aaf84a56c722" path="/var/lib/kubelet/pods/81d7c2b0-44d4-4b9f-8000-aaf84a56c722/volumes"
Oct 31 14:04:30.755912 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:30.755899    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc620c07-7229-4b69-9686-6ae7786b869f" path="/var/lib/kubelet/pods/bc620c07-7229-4b69-9686-6ae7786b869f/volumes"
Oct 31 14:04:31.186939 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:31.186812    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs" podUID="859ad249-88cd-4054-a411-f0847db07b12" containerName="payload-processing" containerID="cri-o://08dc6979326f775489bb6ebcaeb572c9aa2e9aab1dbb3e5fde0e34ed28ab4e9d" gracePeriod=25
Oct 31 14:04:32.042846 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:32.042805    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs" podUID="859ad249-88cd-4054-a411-f0847db07b12" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:04:41.400958 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.400923    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"]
Oct 31 14:04:41.401504 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.401131    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8" podUID="50cf0c64-755b-4440-85b0-2657ea896014" containerName="maas-api" containerID="cri-o://9adaacb7fac9c5a70aba8bd74af5132a7c9044fc2c14e0abbd6337258172550f" gracePeriod=30
Oct 31 14:04:41.476756 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.476718    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8" podUID="50cf0c64-755b-4440-85b0-2657ea896014" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.52:8443/health\": dial tcp 10.134.0.52:8443: connect: connection refused"
Oct 31 14:04:41.506144 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.506113    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:41.657974 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.657941    4428 generic.go:356] "Generic (PLEG): container finished" podID="50cf0c64-755b-4440-85b0-2657ea896014" containerID="9adaacb7fac9c5a70aba8bd74af5132a7c9044fc2c14e0abbd6337258172550f" exitCode=0
Oct 31 14:04:41.658135 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.658019    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8" event={"ID":"50cf0c64-755b-4440-85b0-2657ea896014","Type":"ContainerDied","Data":"9adaacb7fac9c5a70aba8bd74af5132a7c9044fc2c14e0abbd6337258172550f"}
Oct 31 14:04:41.658135 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.658081    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8" event={"ID":"50cf0c64-755b-4440-85b0-2657ea896014","Type":"ContainerDied","Data":"dcb0fbde9ad49fc445bf9090813fda2732c3bb82894195c9ad593d6f478f6fdb"}
Oct 31 14:04:41.658135 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.658109    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcb0fbde9ad49fc445bf9090813fda2732c3bb82894195c9ad593d6f478f6fdb"
Oct 31 14:04:41.673613 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.673543    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 14:04:41.809609 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.809573    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50cf0c64-755b-4440-85b0-2657ea896014-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cf0c64-755b-4440-85b0-2657ea896014-openshift-service-ca\") pod \"50cf0c64-755b-4440-85b0-2657ea896014\" (UID: \"50cf0c64-755b-4440-85b0-2657ea896014\") "
Oct 31 14:04:41.809814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.809657    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cf0c64-755b-4440-85b0-2657ea896014-kube-api-access-55m6v\" (UniqueName: \"kubernetes.io/projected/50cf0c64-755b-4440-85b0-2657ea896014-kube-api-access-55m6v\") pod \"50cf0c64-755b-4440-85b0-2657ea896014\" (UID: \"50cf0c64-755b-4440-85b0-2657ea896014\") "
Oct 31 14:04:41.809814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.809755    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50cf0c64-755b-4440-85b0-2657ea896014-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50cf0c64-755b-4440-85b0-2657ea896014-metrics-certs\") pod \"50cf0c64-755b-4440-85b0-2657ea896014\" (UID: \"50cf0c64-755b-4440-85b0-2657ea896014\") "
Oct 31 14:04:41.809997 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.809842    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50cf0c64-755b-4440-85b0-2657ea896014-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50cf0c64-755b-4440-85b0-2657ea896014-maas-api-tls\") pod \"50cf0c64-755b-4440-85b0-2657ea896014\" (UID: \"50cf0c64-755b-4440-85b0-2657ea896014\") "
Oct 31 14:04:41.809997 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.809982    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cf0c64-755b-4440-85b0-2657ea896014-openshift-service-ca" pod "50cf0c64-755b-4440-85b0-2657ea896014" (UID: "50cf0c64-755b-4440-85b0-2657ea896014"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:04:41.810708 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.810440    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cf0c64-755b-4440-85b0-2657ea896014-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:41.812137 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.812102    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50cf0c64-755b-4440-85b0-2657ea896014-metrics-certs" pod "50cf0c64-755b-4440-85b0-2657ea896014" (UID: "50cf0c64-755b-4440-85b0-2657ea896014"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:04:41.812137 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.812140    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50cf0c64-755b-4440-85b0-2657ea896014-maas-api-tls" pod "50cf0c64-755b-4440-85b0-2657ea896014" (UID: "50cf0c64-755b-4440-85b0-2657ea896014"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:04:41.812313 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.812223    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cf0c64-755b-4440-85b0-2657ea896014-kube-api-access-55m6v" pod "50cf0c64-755b-4440-85b0-2657ea896014" (UID: "50cf0c64-755b-4440-85b0-2657ea896014"). InnerVolumeSpecName "kube-api-access-55m6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:04:41.911186 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.911100    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55m6v\" (UniqueName: \"kubernetes.io/projected/50cf0c64-755b-4440-85b0-2657ea896014-kube-api-access-55m6v\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:41.911186 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.911127    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50cf0c64-755b-4440-85b0-2657ea896014-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:41.911186 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:41.911137    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50cf0c64-755b-4440-85b0-2657ea896014-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:42.042468 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:42.042423    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs" podUID="859ad249-88cd-4054-a411-f0847db07b12" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:04:42.673130 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:42.673086    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"
Oct 31 14:04:42.699062 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:42.699024    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"]
Oct 31 14:04:42.705941 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:42.705904    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8b2f0a-5dbcc8f5cb-7nnf8"]
Oct 31 14:04:42.742793 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:42.742758    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:42.753565 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:42.753524    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50cf0c64-755b-4440-85b0-2657ea896014" path="/var/lib/kubelet/pods/50cf0c64-755b-4440-85b0-2657ea896014/volumes"
Oct 31 14:04:48.741382 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:04:48.741341    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:04:49.588148 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:49.587495    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:49.743756 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:49.743721    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:49.850577 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:49.850474    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"]
Oct 31 14:04:49.850796 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:49.850771    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h" podUID="a7c64ba1-f422-427b-80b8-1f464d32fafe" containerName="istio-proxy" containerID="cri-o://e5971e16f8ebf07d4b0aa829efb4b5d4c3f4e6520f11802d7cfdc36d1ef4012a" gracePeriod=30
Oct 31 14:04:49.937586 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:49.937541    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:50.874668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:50.874618    4428 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:04:50.875169 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:50.874700    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h" podUID="a7c64ba1-f422-427b-80b8-1f464d32fafe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:04:51.442203 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:51.442177    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"
Oct 31 14:04:51.613296 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:51.613258    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/859ad249-88cd-4054-a411-f0847db07b12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859ad249-88cd-4054-a411-f0847db07b12-plugins-config-volume\") pod \"859ad249-88cd-4054-a411-f0847db07b12\" (UID: \"859ad249-88cd-4054-a411-f0847db07b12\") "
Oct 31 14:04:51.613526 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:51.613350    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/859ad249-88cd-4054-a411-f0847db07b12-kube-api-access-kl46z\" (UniqueName: \"kubernetes.io/projected/859ad249-88cd-4054-a411-f0847db07b12-kube-api-access-kl46z\") pod \"859ad249-88cd-4054-a411-f0847db07b12\" (UID: \"859ad249-88cd-4054-a411-f0847db07b12\") "
Oct 31 14:04:51.613726 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:51.613701    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/859ad249-88cd-4054-a411-f0847db07b12-plugins-config-volume" pod "859ad249-88cd-4054-a411-f0847db07b12" (UID: "859ad249-88cd-4054-a411-f0847db07b12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:04:51.615664 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:51.615638    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859ad249-88cd-4054-a411-f0847db07b12-kube-api-access-kl46z" pod "859ad249-88cd-4054-a411-f0847db07b12" (UID: "859ad249-88cd-4054-a411-f0847db07b12"). InnerVolumeSpecName "kube-api-access-kl46z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:04:51.715147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:51.715111    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859ad249-88cd-4054-a411-f0847db07b12-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:51.715147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:51.715142    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl46z\" (UniqueName: \"kubernetes.io/projected/859ad249-88cd-4054-a411-f0847db07b12-kube-api-access-kl46z\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:51.725479 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:51.725448    4428 generic.go:356] "Generic (PLEG): container finished" podID="859ad249-88cd-4054-a411-f0847db07b12" containerID="08dc6979326f775489bb6ebcaeb572c9aa2e9aab1dbb3e5fde0e34ed28ab4e9d" exitCode=0
Oct 31 14:04:51.725657 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:51.725497    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"
Oct 31 14:04:51.725657 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:51.725534    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs" event={"ID":"859ad249-88cd-4054-a411-f0847db07b12","Type":"ContainerDied","Data":"08dc6979326f775489bb6ebcaeb572c9aa2e9aab1dbb3e5fde0e34ed28ab4e9d"}
Oct 31 14:04:51.725657 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:51.725594    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs" event={"ID":"859ad249-88cd-4054-a411-f0847db07b12","Type":"ContainerDied","Data":"b982a932440f42b347622d9ad123e4cc806e6ad7718d41ae13194f59f1a4888c"}
Oct 31 14:04:51.725657 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:51.725622    4428 scope.go:122] "RemoveContainer" containerID="08dc6979326f775489bb6ebcaeb572c9aa2e9aab1dbb3e5fde0e34ed28ab4e9d"
Oct 31 14:04:51.735648 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:51.735629    4428 scope.go:122] "RemoveContainer" containerID="08dc6979326f775489bb6ebcaeb572c9aa2e9aab1dbb3e5fde0e34ed28ab4e9d"
Oct 31 14:04:51.735937 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:04:51.735914    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08dc6979326f775489bb6ebcaeb572c9aa2e9aab1dbb3e5fde0e34ed28ab4e9d\": container with ID starting with 08dc6979326f775489bb6ebcaeb572c9aa2e9aab1dbb3e5fde0e34ed28ab4e9d not found: ID does not exist" containerID="08dc6979326f775489bb6ebcaeb572c9aa2e9aab1dbb3e5fde0e34ed28ab4e9d"
Oct 31 14:04:51.736058 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:51.735945    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08dc6979326f775489bb6ebcaeb572c9aa2e9aab1dbb3e5fde0e34ed28ab4e9d"} err="failed to get container status \"08dc6979326f775489bb6ebcaeb572c9aa2e9aab1dbb3e5fde0e34ed28ab4e9d\": rpc error: code = NotFound desc = could not find container \"08dc6979326f775489bb6ebcaeb572c9aa2e9aab1dbb3e5fde0e34ed28ab4e9d\": container with ID starting with 08dc6979326f775489bb6ebcaeb572c9aa2e9aab1dbb3e5fde0e34ed28ab4e9d not found: ID does not exist"
Oct 31 14:04:51.759947 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:51.759914    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"]
Oct 31 14:04:51.775148 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:51.775118    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8b2f0a-544f99cf49-r4qxs"]
Oct 31 14:04:52.753659 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:52.753628    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="859ad249-88cd-4054-a411-f0847db07b12" path="/var/lib/kubelet/pods/859ad249-88cd-4054-a411-f0847db07b12/volumes"
Oct 31 14:04:55.222841 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.222815    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 14:04:55.351723 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.351669    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-workload-socket\") pod \"a7c64ba1-f422-427b-80b8-1f464d32fafe\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") "
Oct 31 14:04:55.351978 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.351790    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c64ba1-f422-427b-80b8-1f464d32fafe-kube-api-access-7bfs5\" (UniqueName: \"kubernetes.io/projected/a7c64ba1-f422-427b-80b8-1f464d32fafe-kube-api-access-7bfs5\") pod \"a7c64ba1-f422-427b-80b8-1f464d32fafe\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") "
Oct 31 14:04:55.351978 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.351893    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-podinfo\") pod \"a7c64ba1-f422-427b-80b8-1f464d32fafe\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") "
Oct 31 14:04:55.351978 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.351927    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c64ba1-f422-427b-80b8-1f464d32fafe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c64ba1-f422-427b-80b8-1f464d32fafe-istiod-ca-cert\") pod \"a7c64ba1-f422-427b-80b8-1f464d32fafe\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") "
Oct 31 14:04:55.351978 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.351958    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-data\") pod \"a7c64ba1-f422-427b-80b8-1f464d32fafe\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") "
Oct 31 14:04:55.352483 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.352030    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-token\" (UniqueName: \"kubernetes.io/projected/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-token\") pod \"a7c64ba1-f422-427b-80b8-1f464d32fafe\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") "
Oct 31 14:04:55.352483 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.352093    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-envoy\") pod \"a7c64ba1-f422-427b-80b8-1f464d32fafe\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") "
Oct 31 14:04:55.352483 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.352162    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-credential-socket\") pod \"a7c64ba1-f422-427b-80b8-1f464d32fafe\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") "
Oct 31 14:04:55.352483 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.352198    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-workload-certs\") pod \"a7c64ba1-f422-427b-80b8-1f464d32fafe\" (UID: \"a7c64ba1-f422-427b-80b8-1f464d32fafe\") "
Oct 31 14:04:55.353104 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.352813    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-workload-certs" pod "a7c64ba1-f422-427b-80b8-1f464d32fafe" (UID: "a7c64ba1-f422-427b-80b8-1f464d32fafe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:55.353629 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.353414    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c64ba1-f422-427b-80b8-1f464d32fafe-istiod-ca-cert" pod "a7c64ba1-f422-427b-80b8-1f464d32fafe" (UID: "a7c64ba1-f422-427b-80b8-1f464d32fafe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:04:55.353809 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.353756    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-workload-socket" pod "a7c64ba1-f422-427b-80b8-1f464d32fafe" (UID: "a7c64ba1-f422-427b-80b8-1f464d32fafe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:55.355781 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.355515    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-data" pod "a7c64ba1-f422-427b-80b8-1f464d32fafe" (UID: "a7c64ba1-f422-427b-80b8-1f464d32fafe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:55.356270 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.356096    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-credential-socket" pod "a7c64ba1-f422-427b-80b8-1f464d32fafe" (UID: "a7c64ba1-f422-427b-80b8-1f464d32fafe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:55.363484 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.363407    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-token" pod "a7c64ba1-f422-427b-80b8-1f464d32fafe" (UID: "a7c64ba1-f422-427b-80b8-1f464d32fafe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:04:55.363972 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.363933    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-podinfo" pod "a7c64ba1-f422-427b-80b8-1f464d32fafe" (UID: "a7c64ba1-f422-427b-80b8-1f464d32fafe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:04:55.365669 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.365640    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c64ba1-f422-427b-80b8-1f464d32fafe-kube-api-access-7bfs5" pod "a7c64ba1-f422-427b-80b8-1f464d32fafe" (UID: "a7c64ba1-f422-427b-80b8-1f464d32fafe"). InnerVolumeSpecName "kube-api-access-7bfs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:04:55.366115 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.366090    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-envoy" pod "a7c64ba1-f422-427b-80b8-1f464d32fafe" (UID: "a7c64ba1-f422-427b-80b8-1f464d32fafe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:04:55.454122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.454003    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.454122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.454051    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.454122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.454069    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.454122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.454086    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.454122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.454101    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.454122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.454117    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bfs5\" (UniqueName: \"kubernetes.io/projected/a7c64ba1-f422-427b-80b8-1f464d32fafe-kube-api-access-7bfs5\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.454660 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.454134    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.454660 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.454150    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c64ba1-f422-427b-80b8-1f464d32fafe-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.454660 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.454167    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c64ba1-f422-427b-80b8-1f464d32fafe-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:04:55.752470 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.752252    4428 generic.go:356] "Generic (PLEG): container finished" podID="a7c64ba1-f422-427b-80b8-1f464d32fafe" containerID="e5971e16f8ebf07d4b0aa829efb4b5d4c3f4e6520f11802d7cfdc36d1ef4012a" exitCode=0
Oct 31 14:04:55.752470 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.752292    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h" event={"ID":"a7c64ba1-f422-427b-80b8-1f464d32fafe","Type":"ContainerDied","Data":"e5971e16f8ebf07d4b0aa829efb4b5d4c3f4e6520f11802d7cfdc36d1ef4012a"}
Oct 31 14:04:55.752470 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.752316    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"
Oct 31 14:04:55.752470 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.752342    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h" event={"ID":"a7c64ba1-f422-427b-80b8-1f464d32fafe","Type":"ContainerDied","Data":"669145f1fca1dcf30168f058f873faa47ad1120b52aa0520bf74a08e20b60809"}
Oct 31 14:04:55.752470 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.752363    4428 scope.go:122] "RemoveContainer" containerID="e5971e16f8ebf07d4b0aa829efb4b5d4c3f4e6520f11802d7cfdc36d1ef4012a"
Oct 31 14:04:55.765535 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.765432    4428 scope.go:122] "RemoveContainer" containerID="e5971e16f8ebf07d4b0aa829efb4b5d4c3f4e6520f11802d7cfdc36d1ef4012a"
Oct 31 14:04:55.766308 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:04:55.766263    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5971e16f8ebf07d4b0aa829efb4b5d4c3f4e6520f11802d7cfdc36d1ef4012a\": container with ID starting with e5971e16f8ebf07d4b0aa829efb4b5d4c3f4e6520f11802d7cfdc36d1ef4012a not found: ID does not exist" containerID="e5971e16f8ebf07d4b0aa829efb4b5d4c3f4e6520f11802d7cfdc36d1ef4012a"
Oct 31 14:04:55.766422 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.766314    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5971e16f8ebf07d4b0aa829efb4b5d4c3f4e6520f11802d7cfdc36d1ef4012a"} err="failed to get container status \"e5971e16f8ebf07d4b0aa829efb4b5d4c3f4e6520f11802d7cfdc36d1ef4012a\": rpc error: code = NotFound desc = could not find container \"e5971e16f8ebf07d4b0aa829efb4b5d4c3f4e6520f11802d7cfdc36d1ef4012a\": container with ID starting with e5971e16f8ebf07d4b0aa829efb4b5d4c3f4e6520f11802d7cfdc36d1ef4012a not found: ID does not exist"
Oct 31 14:04:55.782845 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.782808    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"]
Oct 31 14:04:55.791768 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:55.791728    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-rbb6h"]
Oct 31 14:04:56.754280 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:56.754248    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c64ba1-f422-427b-80b8-1f464d32fafe" path="/var/lib/kubelet/pods/a7c64ba1-f422-427b-80b8-1f464d32fafe/volumes"
Oct 31 14:04:58.011980 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:58.011938    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:04:58.189721 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:04:58.189683    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:05:00.009518 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:00.009480    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:05:05.989430 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:05.989387    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:05:13.857171 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857130    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"]
Oct 31 14:05:13.857628 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857564    4428 cpu_manager.go:414] "containerMap: removing container" podUID="859ad249-88cd-4054-a411-f0847db07b12" containerName="payload-processing"
Oct 31 14:05:13.857628 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857584    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="859ad249-88cd-4054-a411-f0847db07b12" containerName="payload-processing"
Oct 31 14:05:13.857628 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857596    4428 cpu_manager.go:414] "containerMap: removing container" podUID="81d7c2b0-44d4-4b9f-8000-aaf84a56c722" containerName="payload-processing"
Oct 31 14:05:13.857628 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857601    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d7c2b0-44d4-4b9f-8000-aaf84a56c722" containerName="payload-processing"
Oct 31 14:05:13.857628 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857623    4428 cpu_manager.go:414] "containerMap: removing container" podUID="5701fca5-38a3-4a28-86b3-55124cb6f54e" containerName="istio-proxy"
Oct 31 14:05:13.857628 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857628    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5701fca5-38a3-4a28-86b3-55124cb6f54e" containerName="istio-proxy"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857642    4428 cpu_manager.go:414] "containerMap: removing container" podUID="a7c64ba1-f422-427b-80b8-1f464d32fafe" containerName="istio-proxy"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857650    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c64ba1-f422-427b-80b8-1f464d32fafe" containerName="istio-proxy"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857661    4428 cpu_manager.go:414] "containerMap: removing container" podUID="bc620c07-7229-4b69-9686-6ae7786b869f" containerName="payload-pre-processing"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857669    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc620c07-7229-4b69-9686-6ae7786b869f" containerName="payload-pre-processing"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857682    4428 cpu_manager.go:414] "containerMap: removing container" podUID="2d678e30-767d-48e8-8c7a-74c31e08ed30" containerName="payload-pre-processing"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857689    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d678e30-767d-48e8-8c7a-74c31e08ed30" containerName="payload-pre-processing"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857698    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f" containerName="revoke-keys"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857705    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f" containerName="revoke-keys"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857715    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" containerName="istio-proxy"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857721    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" containerName="istio-proxy"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857730    4428 cpu_manager.go:414] "containerMap: removing container" podUID="50cf0c64-755b-4440-85b0-2657ea896014" containerName="maas-api"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857736    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50cf0c64-755b-4440-85b0-2657ea896014" containerName="maas-api"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857829    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e6e17a-543e-4373-ad93-e0ed60c74334" containerName="istio-proxy"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857841    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc620c07-7229-4b69-9686-6ae7786b869f" containerName="payload-pre-processing"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857847    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d678e30-767d-48e8-8c7a-74c31e08ed30" containerName="payload-pre-processing"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857853    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37ae6e0-4b21-4d4d-b8c9-c49a68ebec2f" containerName="revoke-keys"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857860    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd0ff63-3dca-4fc1-b8b1-9bbd7b3438af" containerName="istio-proxy"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857883    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="50cf0c64-755b-4440-85b0-2657ea896014" containerName="maas-api"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857894    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="859ad249-88cd-4054-a411-f0847db07b12" containerName="payload-processing"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857904    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d7c2b0-44d4-4b9f-8000-aaf84a56c722" containerName="payload-processing"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857916    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5701fca5-38a3-4a28-86b3-55124cb6f54e" containerName="istio-proxy"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857923    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5701fca5-38a3-4a28-86b3-55124cb6f54e" containerName="istio-proxy"
Oct 31 14:05:13.857919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.857929    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c64ba1-f422-427b-80b8-1f464d32fafe" containerName="istio-proxy"
Oct 31 14:05:13.861715 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.861603    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:13.877411 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.877382    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"]
Oct 31 14:05:13.911421 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.911386    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:13.911421 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.911432    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:13.911634 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.911478    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:13.911634 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.911545    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28fhj\" (UniqueName: \"kubernetes.io/projected/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-kube-api-access-28fhj\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:13.911634 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.911604    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:13.911634 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.911632    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:13.911813 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.911651    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:13.911813 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.911672    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:13.911813 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:13.911758    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.013096 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.013055    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.013304 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.013117    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.013304 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.013165    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.013304 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.013188    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.013304 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.013212    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.013548 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.013364    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28fhj\" (UniqueName: \"kubernetes.io/projected/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-kube-api-access-28fhj\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.013548 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.013415    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.013548 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.013458    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.013548 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.013490    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.013780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.013561    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.013780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.013662    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.013780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.013690    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.013986 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.013819    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.014044 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.013986    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.015914 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.015852    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.016039 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.015895    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.027331 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.027304    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28fhj\" (UniqueName: \"kubernetes.io/projected/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-kube-api-access-28fhj\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.027435 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.027375    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa86ed88-56ab-4fc4-b6a4-bc7732a8db25-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-7sz9s\" (UID: \"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.113724 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.113641    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"]
Oct 31 14:05:14.114111 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.114095    4428 cpu_manager.go:414] "containerMap: removing container" podUID="5701fca5-38a3-4a28-86b3-55124cb6f54e" containerName="istio-proxy"
Oct 31 14:05:14.114187 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.114115    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5701fca5-38a3-4a28-86b3-55124cb6f54e" containerName="istio-proxy"
Oct 31 14:05:14.118428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.118411    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"
Oct 31 14:05:14.124937 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.124910    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:05:14.125170 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.125119    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:05:14.128192 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.128164    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"]
Oct 31 14:05:14.176581 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.176554    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:14.215398 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.215357    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2klj\" (UniqueName: \"kubernetes.io/projected/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-kube-api-access-v2klj\") pod \"maas-api-revoke-sub-85237e45-cn92m\" (UID: \"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"
Oct 31 14:05:14.215581 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.215416    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-cn92m\" (UID: \"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"
Oct 31 14:05:14.215627 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.215575    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-cn92m\" (UID: \"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"
Oct 31 14:05:14.316649 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.316611    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-cn92m\" (UID: \"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"
Oct 31 14:05:14.316850 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.316700    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2klj\" (UniqueName: \"kubernetes.io/projected/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-kube-api-access-v2klj\") pod \"maas-api-revoke-sub-85237e45-cn92m\" (UID: \"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"
Oct 31 14:05:14.316850 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.316731    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-cn92m\" (UID: \"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"
Oct 31 14:05:14.317522 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.317459    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-cn92m\" (UID: \"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"
Oct 31 14:05:14.319292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.319268    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-cn92m\" (UID: \"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"
Oct 31 14:05:14.331162 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.331135    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2klj\" (UniqueName: \"kubernetes.io/projected/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-kube-api-access-v2klj\") pod \"maas-api-revoke-sub-85237e45-cn92m\" (UID: \"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"
Oct 31 14:05:14.434974 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.434899    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"
Oct 31 14:05:14.463734 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.463697    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"]
Oct 31 14:05:14.465892 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:05:14.465850    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa86ed88_56ab_4fc4_b6a4_bc7732a8db25.slice/crio-8b1e1f297432ff13ddfa3312472b9d7c410796db311dc1d12324991a979935f9 WatchSource:0}: Error finding container 8b1e1f297432ff13ddfa3312472b9d7c410796db311dc1d12324991a979935f9: Status 404 returned error can't find the container with id 8b1e1f297432ff13ddfa3312472b9d7c410796db311dc1d12324991a979935f9
Oct 31 14:05:14.468706 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.468672    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:05:14.468788 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.468743    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:05:14.468788 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.468774    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:05:14.723381 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:05:14.723343    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd060dc2b_6965_4813_9dc4_c9c42c1e9c3c.slice/crio-b2742c4ad6c8ed79493fc2e52dd5cd9789e9dab15af223def245ff488f12c774 WatchSource:0}: Error finding container b2742c4ad6c8ed79493fc2e52dd5cd9789e9dab15af223def245ff488f12c774: Status 404 returned error can't find the container with id b2742c4ad6c8ed79493fc2e52dd5cd9789e9dab15af223def245ff488f12c774
Oct 31 14:05:14.729521 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.729356    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"]
Oct 31 14:05:14.858110 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.858066    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s" event={"ID":"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25","Type":"ContainerStarted","Data":"2d561808d351d75978e63cea32fb06491e8ccf3d186d55252d0cadf207784b50"}
Oct 31 14:05:14.858643 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.858136    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s" event={"ID":"aa86ed88-56ab-4fc4-b6a4-bc7732a8db25","Type":"ContainerStarted","Data":"8b1e1f297432ff13ddfa3312472b9d7c410796db311dc1d12324991a979935f9"}
Oct 31 14:05:14.859414 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.859390    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m" event={"ID":"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c","Type":"ContainerStarted","Data":"b2742c4ad6c8ed79493fc2e52dd5cd9789e9dab15af223def245ff488f12c774"}
Oct 31 14:05:14.884032 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:14.883976    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s" podStartSLOduration=1.88395671 podStartE2EDuration="1.88395671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:05:14.881519731 +0000 UTC m=+3712.279868356" watchObservedRunningTime="2026-10-31 14:05:14.88395671 +0000 UTC m=+3712.282305331"
Oct 31 14:05:15.177246 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:15.177207    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:15.179172 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:15.179140    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body=
Oct 31 14:05:15.179334 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:15.179193    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s" podUID="aa86ed88-56ab-4fc4-b6a4-bc7732a8db25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused"
Oct 31 14:05:15.878390 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:15.878348    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m" event={"ID":"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c","Type":"ContainerStarted","Data":"d570083bcc6bfb40e6bd93fc81826f7f174949e9b756ace73ee0bd4af6698533"}
Oct 31 14:05:15.892420 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:15.892391    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:05:16.177700 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:16.177589    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body=
Oct 31 14:05:16.177700 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:16.177668    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s" podUID="aa86ed88-56ab-4fc4-b6a4-bc7732a8db25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused"
Oct 31 14:05:16.599148 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:16.599114    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:05:16.885079 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:16.884982    4428 generic.go:356] "Generic (PLEG): container finished" podID="d060dc2b-6965-4813-9dc4-c9c42c1e9c3c" containerID="d570083bcc6bfb40e6bd93fc81826f7f174949e9b756ace73ee0bd4af6698533" exitCode=0
Oct 31 14:05:16.885079 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:16.885065    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m" event={"ID":"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c","Type":"ContainerDied","Data":"d570083bcc6bfb40e6bd93fc81826f7f174949e9b756ace73ee0bd4af6698533"}
Oct 31 14:05:17.031020 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:17.030992    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"
Oct 31 14:05:17.144408 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:17.144311    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-maas-api-openshift-ca\") pod \"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c\" (UID: \"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c\") "
Oct 31 14:05:17.144408 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:17.144407    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-maas-api-xks-ca\") pod \"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c\" (UID: \"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c\") "
Oct 31 14:05:17.144689 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:17.144474    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-kube-api-access-v2klj\" (UniqueName: \"kubernetes.io/projected/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-kube-api-access-v2klj\") pod \"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c\" (UID: \"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c\") "
Oct 31 14:05:17.144762 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:17.144736    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-maas-api-openshift-ca" pod "d060dc2b-6965-4813-9dc4-c9c42c1e9c3c" (UID: "d060dc2b-6965-4813-9dc4-c9c42c1e9c3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:05:17.171122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:17.171071    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-maas-api-xks-ca" pod "d060dc2b-6965-4813-9dc4-c9c42c1e9c3c" (UID: "d060dc2b-6965-4813-9dc4-c9c42c1e9c3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:05:17.171477 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:17.171436    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-kube-api-access-v2klj" pod "d060dc2b-6965-4813-9dc4-c9c42c1e9c3c" (UID: "d060dc2b-6965-4813-9dc4-c9c42c1e9c3c"). InnerVolumeSpecName "kube-api-access-v2klj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:05:17.177956 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:17.177923    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body=
Oct 31 14:05:17.178076 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:17.177989    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s" podUID="aa86ed88-56ab-4fc4-b6a4-bc7732a8db25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused"
Oct 31 14:05:17.264895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:17.252512    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:05:17.264895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:17.252560    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2klj\" (UniqueName: \"kubernetes.io/projected/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-kube-api-access-v2klj\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:05:17.264895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:17.252577    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:05:17.891593 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:17.891562    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"
Oct 31 14:05:17.891593 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:17.891586    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m" event={"ID":"d060dc2b-6965-4813-9dc4-c9c42c1e9c3c","Type":"ContainerDied","Data":"b2742c4ad6c8ed79493fc2e52dd5cd9789e9dab15af223def245ff488f12c774"}
Oct 31 14:05:17.892092 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:17.891635    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2742c4ad6c8ed79493fc2e52dd5cd9789e9dab15af223def245ff488f12c774"
Oct 31 14:05:18.177249 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:18.177156    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body=
Oct 31 14:05:18.177249 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:18.177215    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s" podUID="aa86ed88-56ab-4fc4-b6a4-bc7732a8db25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused"
Oct 31 14:05:18.740486 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:05:18.740441    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:05:19.177978 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:19.177883    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body=
Oct 31 14:05:19.177978 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:19.177936    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s" podUID="aa86ed88-56ab-4fc4-b6a4-bc7732a8db25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused"
Oct 31 14:05:20.177469 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:20.177428    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body=
Oct 31 14:05:20.177675 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:20.177502    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s" podUID="aa86ed88-56ab-4fc4-b6a4-bc7732a8db25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused"
Oct 31 14:05:21.177294 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:21.177256    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body=
Oct 31 14:05:21.177730 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:21.177344    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s" podUID="aa86ed88-56ab-4fc4-b6a4-bc7732a8db25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused"
Oct 31 14:05:23.178006 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:23.177958    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 14:05:23.178574 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:23.178035    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s" podUID="aa86ed88-56ab-4fc4-b6a4-bc7732a8db25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 14:05:24.219067 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:24.219021    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 14:05:24.219587 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:24.219091    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s" podUID="aa86ed88-56ab-4fc4-b6a4-bc7732a8db25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 14:05:24.770808 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:24.770771    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:24.771141 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:24.771116    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:24.771833 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:24.771815    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-7sz9s"
Oct 31 14:05:25.747687 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:25.747652    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:05:28.262006 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:28.261972    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:05:35.908475 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:35.908390    4428 scope.go:122] "RemoveContainer" containerID="9adaacb7fac9c5a70aba8bd74af5132a7c9044fc2c14e0abbd6337258172550f"
Oct 31 14:05:37.026258 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:37.026204    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:05:48.742388 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:05:48.742344    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:05:53.707951 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:53.707918    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:05:53.863172 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:53.863139    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:05:54.026310 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:54.026274    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"]
Oct 31 14:05:54.029929 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:54.029901    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b0a7360-4h5d2"]
Oct 31 14:05:54.753502 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:05:54.753466    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f431fba-c1ad-4520-899e-774be2a5d007" path="/var/lib/kubelet/pods/4f431fba-c1ad-4520-899e-774be2a5d007/volumes"
Oct 31 14:06:05.904714 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:05.904675    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"]
Oct 31 14:06:05.905185 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:05.905097    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d060dc2b-6965-4813-9dc4-c9c42c1e9c3c" containerName="revoke-keys"
Oct 31 14:06:05.905185 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:05.905108    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d060dc2b-6965-4813-9dc4-c9c42c1e9c3c" containerName="revoke-keys"
Oct 31 14:06:05.905185 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:05.905180    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d060dc2b-6965-4813-9dc4-c9c42c1e9c3c" containerName="revoke-keys"
Oct 31 14:06:05.908403 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:05.908385    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"
Oct 31 14:06:05.916426 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:05.916400    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:06:05.916567 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:05.916465    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:06:05.918794 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:05.918763    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"]
Oct 31 14:06:05.972155 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:05.972114    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47a9c61f-10c8-449f-9dd1-39b13c846c5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tvpjd\" (UID: \"47a9c61f-10c8-449f-9dd1-39b13c846c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"
Oct 31 14:06:05.972348 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:05.972176    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk6dr\" (UniqueName: \"kubernetes.io/projected/47a9c61f-10c8-449f-9dd1-39b13c846c5e-kube-api-access-rk6dr\") pod \"maas-api-revoke-sub-c1e271a9-tvpjd\" (UID: \"47a9c61f-10c8-449f-9dd1-39b13c846c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"
Oct 31 14:06:05.972397 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:05.972367    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47a9c61f-10c8-449f-9dd1-39b13c846c5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tvpjd\" (UID: \"47a9c61f-10c8-449f-9dd1-39b13c846c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"
Oct 31 14:06:06.073426 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:06.073380    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6dr\" (UniqueName: \"kubernetes.io/projected/47a9c61f-10c8-449f-9dd1-39b13c846c5e-kube-api-access-rk6dr\") pod \"maas-api-revoke-sub-c1e271a9-tvpjd\" (UID: \"47a9c61f-10c8-449f-9dd1-39b13c846c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"
Oct 31 14:06:06.073629 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:06.073468    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47a9c61f-10c8-449f-9dd1-39b13c846c5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tvpjd\" (UID: \"47a9c61f-10c8-449f-9dd1-39b13c846c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"
Oct 31 14:06:06.073629 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:06.073504    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47a9c61f-10c8-449f-9dd1-39b13c846c5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tvpjd\" (UID: \"47a9c61f-10c8-449f-9dd1-39b13c846c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"
Oct 31 14:06:06.074312 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:06.074236    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47a9c61f-10c8-449f-9dd1-39b13c846c5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tvpjd\" (UID: \"47a9c61f-10c8-449f-9dd1-39b13c846c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"
Oct 31 14:06:06.076133 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:06.076099    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47a9c61f-10c8-449f-9dd1-39b13c846c5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tvpjd\" (UID: \"47a9c61f-10c8-449f-9dd1-39b13c846c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"
Oct 31 14:06:06.087949 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:06.087921    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk6dr\" (UniqueName: \"kubernetes.io/projected/47a9c61f-10c8-449f-9dd1-39b13c846c5e-kube-api-access-rk6dr\") pod \"maas-api-revoke-sub-c1e271a9-tvpjd\" (UID: \"47a9c61f-10c8-449f-9dd1-39b13c846c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"
Oct 31 14:06:06.223923 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:06.223862    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"
Oct 31 14:06:06.506262 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:06.506234    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"]
Oct 31 14:06:06.509082 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:06:06.509050    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a9c61f_10c8_449f_9dd1_39b13c846c5e.slice/crio-fa58002ea4ef2d62bfd1e3a4f8b99503f6fd610af37151899ac1de9450ee8ba9 WatchSource:0}: Error finding container fa58002ea4ef2d62bfd1e3a4f8b99503f6fd610af37151899ac1de9450ee8ba9: Status 404 returned error can't find the container with id fa58002ea4ef2d62bfd1e3a4f8b99503f6fd610af37151899ac1de9450ee8ba9
Oct 31 14:06:07.158019 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:07.157925    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd" event={"ID":"47a9c61f-10c8-449f-9dd1-39b13c846c5e","Type":"ContainerStarted","Data":"d314738e96c05b1452593c62cf4077106e72bc73c315b1eb2f77e3320f0e06ac"}
Oct 31 14:06:07.158019 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:07.157986    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd" event={"ID":"47a9c61f-10c8-449f-9dd1-39b13c846c5e","Type":"ContainerStarted","Data":"fa58002ea4ef2d62bfd1e3a4f8b99503f6fd610af37151899ac1de9450ee8ba9"}
Oct 31 14:06:07.185632 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:07.185574    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd" podStartSLOduration=2.185560048 podStartE2EDuration="2.185560048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:06:07.183325144 +0000 UTC m=+3764.581673766" watchObservedRunningTime="2026-10-31 14:06:07.185560048 +0000 UTC m=+3764.583908673"
Oct 31 14:06:08.165488 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:08.165454    4428 generic.go:356] "Generic (PLEG): container finished" podID="47a9c61f-10c8-449f-9dd1-39b13c846c5e" containerID="d314738e96c05b1452593c62cf4077106e72bc73c315b1eb2f77e3320f0e06ac" exitCode=0
Oct 31 14:06:08.166074 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:08.165544    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd" event={"ID":"47a9c61f-10c8-449f-9dd1-39b13c846c5e","Type":"ContainerDied","Data":"d314738e96c05b1452593c62cf4077106e72bc73c315b1eb2f77e3320f0e06ac"}
Oct 31 14:06:09.314538 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:09.314512    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"
Oct 31 14:06:09.405748 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:09.405713    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47a9c61f-10c8-449f-9dd1-39b13c846c5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47a9c61f-10c8-449f-9dd1-39b13c846c5e-maas-api-openshift-ca\") pod \"47a9c61f-10c8-449f-9dd1-39b13c846c5e\" (UID: \"47a9c61f-10c8-449f-9dd1-39b13c846c5e\") "
Oct 31 14:06:09.405980 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:09.405820    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47a9c61f-10c8-449f-9dd1-39b13c846c5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47a9c61f-10c8-449f-9dd1-39b13c846c5e-maas-api-xks-ca\") pod \"47a9c61f-10c8-449f-9dd1-39b13c846c5e\" (UID: \"47a9c61f-10c8-449f-9dd1-39b13c846c5e\") "
Oct 31 14:06:09.405980 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:09.405945    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a9c61f-10c8-449f-9dd1-39b13c846c5e-kube-api-access-rk6dr\" (UniqueName: \"kubernetes.io/projected/47a9c61f-10c8-449f-9dd1-39b13c846c5e-kube-api-access-rk6dr\") pod \"47a9c61f-10c8-449f-9dd1-39b13c846c5e\" (UID: \"47a9c61f-10c8-449f-9dd1-39b13c846c5e\") "
Oct 31 14:06:09.406141 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:09.406089    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a9c61f-10c8-449f-9dd1-39b13c846c5e-maas-api-openshift-ca" pod "47a9c61f-10c8-449f-9dd1-39b13c846c5e" (UID: "47a9c61f-10c8-449f-9dd1-39b13c846c5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:06:09.406239 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:09.406224    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47a9c61f-10c8-449f-9dd1-39b13c846c5e-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:06:09.408221 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:09.408181    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47a9c61f-10c8-449f-9dd1-39b13c846c5e-maas-api-xks-ca" pod "47a9c61f-10c8-449f-9dd1-39b13c846c5e" (UID: "47a9c61f-10c8-449f-9dd1-39b13c846c5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:06:09.408221 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:09.408197    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a9c61f-10c8-449f-9dd1-39b13c846c5e-kube-api-access-rk6dr" pod "47a9c61f-10c8-449f-9dd1-39b13c846c5e" (UID: "47a9c61f-10c8-449f-9dd1-39b13c846c5e"). InnerVolumeSpecName "kube-api-access-rk6dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:06:09.507181 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:09.507141    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk6dr\" (UniqueName: \"kubernetes.io/projected/47a9c61f-10c8-449f-9dd1-39b13c846c5e-kube-api-access-rk6dr\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:06:09.507181 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:09.507172    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47a9c61f-10c8-449f-9dd1-39b13c846c5e-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:06:10.185178 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:10.185142    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"
Oct 31 14:06:10.185178 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:10.185159    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd" event={"ID":"47a9c61f-10c8-449f-9dd1-39b13c846c5e","Type":"ContainerDied","Data":"fa58002ea4ef2d62bfd1e3a4f8b99503f6fd610af37151899ac1de9450ee8ba9"}
Oct 31 14:06:10.185421 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:10.185232    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa58002ea4ef2d62bfd1e3a4f8b99503f6fd610af37151899ac1de9450ee8ba9"
Oct 31 14:06:15.958035 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:15.957996    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:06:16.652264 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:16.652232    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:06:18.740128 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:06:18.740083    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:06:26.967593 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:26.967554    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"]
Oct 31 14:06:26.968049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:26.968034    4428 cpu_manager.go:414] "containerMap: removing container" podUID="47a9c61f-10c8-449f-9dd1-39b13c846c5e" containerName="revoke-keys"
Oct 31 14:06:26.968106 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:26.968050    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a9c61f-10c8-449f-9dd1-39b13c846c5e" containerName="revoke-keys"
Oct 31 14:06:26.968147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:26.968142    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a9c61f-10c8-449f-9dd1-39b13c846c5e" containerName="revoke-keys"
Oct 31 14:06:26.971505 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:26.971489    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"
Oct 31 14:06:26.979149 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:26.979120    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:06:26.979325 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:26.979173    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:06:26.982476 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:26.982449    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"]
Oct 31 14:06:27.009132 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:27.009088    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkxv9\" (UniqueName: \"kubernetes.io/projected/def6c960-6bfa-4b73-986b-8b525f3baa9f-kube-api-access-tkxv9\") pod \"maas-api-revoke-sub-2e1f5b5f-vdzb7\" (UID: \"def6c960-6bfa-4b73-986b-8b525f3baa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"
Oct 31 14:06:27.009343 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:27.009144    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def6c960-6bfa-4b73-986b-8b525f3baa9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vdzb7\" (UID: \"def6c960-6bfa-4b73-986b-8b525f3baa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"
Oct 31 14:06:27.009343 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:27.009201    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def6c960-6bfa-4b73-986b-8b525f3baa9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vdzb7\" (UID: \"def6c960-6bfa-4b73-986b-8b525f3baa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"
Oct 31 14:06:27.110417 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:27.110375    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def6c960-6bfa-4b73-986b-8b525f3baa9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vdzb7\" (UID: \"def6c960-6bfa-4b73-986b-8b525f3baa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"
Oct 31 14:06:27.110626 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:27.110435    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def6c960-6bfa-4b73-986b-8b525f3baa9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vdzb7\" (UID: \"def6c960-6bfa-4b73-986b-8b525f3baa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"
Oct 31 14:06:27.110626 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:27.110510    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkxv9\" (UniqueName: \"kubernetes.io/projected/def6c960-6bfa-4b73-986b-8b525f3baa9f-kube-api-access-tkxv9\") pod \"maas-api-revoke-sub-2e1f5b5f-vdzb7\" (UID: \"def6c960-6bfa-4b73-986b-8b525f3baa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"
Oct 31 14:06:27.111173 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:27.111138    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def6c960-6bfa-4b73-986b-8b525f3baa9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vdzb7\" (UID: \"def6c960-6bfa-4b73-986b-8b525f3baa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"
Oct 31 14:06:27.113050 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:27.113026    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def6c960-6bfa-4b73-986b-8b525f3baa9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vdzb7\" (UID: \"def6c960-6bfa-4b73-986b-8b525f3baa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"
Oct 31 14:06:27.124552 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:27.124526    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkxv9\" (UniqueName: \"kubernetes.io/projected/def6c960-6bfa-4b73-986b-8b525f3baa9f-kube-api-access-tkxv9\") pod \"maas-api-revoke-sub-2e1f5b5f-vdzb7\" (UID: \"def6c960-6bfa-4b73-986b-8b525f3baa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"
Oct 31 14:06:27.287530 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:27.287439    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"
Oct 31 14:06:27.578441 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:27.578410    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"]
Oct 31 14:06:27.580816 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:06:27.580778    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef6c960_6bfa_4b73_986b_8b525f3baa9f.slice/crio-67f4c9f29352320e06d9df813ec55914b16655a9b6f1d6ac94b2923f76369eb3 WatchSource:0}: Error finding container 67f4c9f29352320e06d9df813ec55914b16655a9b6f1d6ac94b2923f76369eb3: Status 404 returned error can't find the container with id 67f4c9f29352320e06d9df813ec55914b16655a9b6f1d6ac94b2923f76369eb3
Oct 31 14:06:28.297831 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:28.297758    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7" event={"ID":"def6c960-6bfa-4b73-986b-8b525f3baa9f","Type":"ContainerStarted","Data":"91ca0903c37e84160f1d80bd9e9f7b6e59933c85abcd5a2a4211408763d1fe04"}
Oct 31 14:06:28.297831 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:28.297813    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7" event={"ID":"def6c960-6bfa-4b73-986b-8b525f3baa9f","Type":"ContainerStarted","Data":"67f4c9f29352320e06d9df813ec55914b16655a9b6f1d6ac94b2923f76369eb3"}
Oct 31 14:06:28.322348 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:28.322290    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7" podStartSLOduration=2.322275877 podStartE2EDuration="2.322275877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:06:28.32178586 +0000 UTC m=+3785.720134508" watchObservedRunningTime="2026-10-31 14:06:28.322275877 +0000 UTC m=+3785.720624501"
Oct 31 14:06:29.305212 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:29.305171    4428 generic.go:356] "Generic (PLEG): container finished" podID="def6c960-6bfa-4b73-986b-8b525f3baa9f" containerID="91ca0903c37e84160f1d80bd9e9f7b6e59933c85abcd5a2a4211408763d1fe04" exitCode=0
Oct 31 14:06:29.305637 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:29.305255    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7" event={"ID":"def6c960-6bfa-4b73-986b-8b525f3baa9f","Type":"ContainerDied","Data":"91ca0903c37e84160f1d80bd9e9f7b6e59933c85abcd5a2a4211408763d1fe04"}
Oct 31 14:06:30.470228 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:30.470200    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"
Oct 31 14:06:30.540981 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:30.540947    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def6c960-6bfa-4b73-986b-8b525f3baa9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def6c960-6bfa-4b73-986b-8b525f3baa9f-maas-api-openshift-ca\") pod \"def6c960-6bfa-4b73-986b-8b525f3baa9f\" (UID: \"def6c960-6bfa-4b73-986b-8b525f3baa9f\") "
Oct 31 14:06:30.541212 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:30.541048    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def6c960-6bfa-4b73-986b-8b525f3baa9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def6c960-6bfa-4b73-986b-8b525f3baa9f-maas-api-xks-ca\") pod \"def6c960-6bfa-4b73-986b-8b525f3baa9f\" (UID: \"def6c960-6bfa-4b73-986b-8b525f3baa9f\") "
Oct 31 14:06:30.541212 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:30.541084    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def6c960-6bfa-4b73-986b-8b525f3baa9f-kube-api-access-tkxv9\" (UniqueName: \"kubernetes.io/projected/def6c960-6bfa-4b73-986b-8b525f3baa9f-kube-api-access-tkxv9\") pod \"def6c960-6bfa-4b73-986b-8b525f3baa9f\" (UID: \"def6c960-6bfa-4b73-986b-8b525f3baa9f\") "
Oct 31 14:06:30.541398 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:30.541371    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def6c960-6bfa-4b73-986b-8b525f3baa9f-maas-api-openshift-ca" pod "def6c960-6bfa-4b73-986b-8b525f3baa9f" (UID: "def6c960-6bfa-4b73-986b-8b525f3baa9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:06:30.543328 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:30.543300    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def6c960-6bfa-4b73-986b-8b525f3baa9f-maas-api-xks-ca" pod "def6c960-6bfa-4b73-986b-8b525f3baa9f" (UID: "def6c960-6bfa-4b73-986b-8b525f3baa9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:06:30.543454 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:30.543436    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def6c960-6bfa-4b73-986b-8b525f3baa9f-kube-api-access-tkxv9" pod "def6c960-6bfa-4b73-986b-8b525f3baa9f" (UID: "def6c960-6bfa-4b73-986b-8b525f3baa9f"). InnerVolumeSpecName "kube-api-access-tkxv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:06:30.642384 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:30.642288    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def6c960-6bfa-4b73-986b-8b525f3baa9f-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:06:30.642384 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:30.642324    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkxv9\" (UniqueName: \"kubernetes.io/projected/def6c960-6bfa-4b73-986b-8b525f3baa9f-kube-api-access-tkxv9\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:06:30.642384 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:30.642340    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def6c960-6bfa-4b73-986b-8b525f3baa9f-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:06:31.317606 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:31.317570    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7" event={"ID":"def6c960-6bfa-4b73-986b-8b525f3baa9f","Type":"ContainerDied","Data":"67f4c9f29352320e06d9df813ec55914b16655a9b6f1d6ac94b2923f76369eb3"}
Oct 31 14:06:31.317797 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:31.317624    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67f4c9f29352320e06d9df813ec55914b16655a9b6f1d6ac94b2923f76369eb3"
Oct 31 14:06:31.317797 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:31.317581    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"
Oct 31 14:06:35.942932 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:35.942900    4428 scope.go:122] "RemoveContainer" containerID="a8dbb5caf7df85f89c368f5a52c0a26c62506949d6522ecea30b510b3dc3d854"
Oct 31 14:06:37.978606 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:37.978563    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"]
Oct 31 14:06:37.979110 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:37.978978    4428 cpu_manager.go:414] "containerMap: removing container" podUID="def6c960-6bfa-4b73-986b-8b525f3baa9f" containerName="revoke-keys"
Oct 31 14:06:37.979110 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:37.978988    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def6c960-6bfa-4b73-986b-8b525f3baa9f" containerName="revoke-keys"
Oct 31 14:06:37.979110 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:37.979089    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="def6c960-6bfa-4b73-986b-8b525f3baa9f" containerName="revoke-keys"
Oct 31 14:06:37.984032 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:37.983993    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"
Oct 31 14:06:37.992654 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:37.992624    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:06:37.992829 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:37.992665    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:06:37.996381 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:37.996349    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"]
Oct 31 14:06:38.113947 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:38.113899    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc7e2cf-7b12-4746-a474-705c25268e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-xt47v\" (UID: \"0dc7e2cf-7b12-4746-a474-705c25268e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"
Oct 31 14:06:38.114135 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:38.113971    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbbjt\" (UniqueName: \"kubernetes.io/projected/0dc7e2cf-7b12-4746-a474-705c25268e6a-kube-api-access-nbbjt\") pod \"maas-api-revoke-sub-ec843018-xt47v\" (UID: \"0dc7e2cf-7b12-4746-a474-705c25268e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"
Oct 31 14:06:38.114135 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:38.114076    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc7e2cf-7b12-4746-a474-705c25268e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-xt47v\" (UID: \"0dc7e2cf-7b12-4746-a474-705c25268e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"
Oct 31 14:06:38.214770 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:38.214733    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc7e2cf-7b12-4746-a474-705c25268e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-xt47v\" (UID: \"0dc7e2cf-7b12-4746-a474-705c25268e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"
Oct 31 14:06:38.215008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:38.214855    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc7e2cf-7b12-4746-a474-705c25268e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-xt47v\" (UID: \"0dc7e2cf-7b12-4746-a474-705c25268e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"
Oct 31 14:06:38.215008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:38.214928    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbbjt\" (UniqueName: \"kubernetes.io/projected/0dc7e2cf-7b12-4746-a474-705c25268e6a-kube-api-access-nbbjt\") pod \"maas-api-revoke-sub-ec843018-xt47v\" (UID: \"0dc7e2cf-7b12-4746-a474-705c25268e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"
Oct 31 14:06:38.216176 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:38.216037    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc7e2cf-7b12-4746-a474-705c25268e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-xt47v\" (UID: \"0dc7e2cf-7b12-4746-a474-705c25268e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"
Oct 31 14:06:38.217783 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:38.217756    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc7e2cf-7b12-4746-a474-705c25268e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-xt47v\" (UID: \"0dc7e2cf-7b12-4746-a474-705c25268e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"
Oct 31 14:06:38.231963 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:38.231897    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbbjt\" (UniqueName: \"kubernetes.io/projected/0dc7e2cf-7b12-4746-a474-705c25268e6a-kube-api-access-nbbjt\") pod \"maas-api-revoke-sub-ec843018-xt47v\" (UID: \"0dc7e2cf-7b12-4746-a474-705c25268e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"
Oct 31 14:06:38.301030 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:38.300988    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"
Oct 31 14:06:38.596744 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:38.596711    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"]
Oct 31 14:06:38.598687 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:06:38.598654    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc7e2cf_7b12_4746_a474_705c25268e6a.slice/crio-87bce7b49795d8c1868f3a904a866eccb9e7d7398d905c9ddca3b7c86f59e1cb WatchSource:0}: Error finding container 87bce7b49795d8c1868f3a904a866eccb9e7d7398d905c9ddca3b7c86f59e1cb: Status 404 returned error can't find the container with id 87bce7b49795d8c1868f3a904a866eccb9e7d7398d905c9ddca3b7c86f59e1cb
Oct 31 14:06:39.366475 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:39.366434    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v" event={"ID":"0dc7e2cf-7b12-4746-a474-705c25268e6a","Type":"ContainerStarted","Data":"6fd7483e8ecb12f8b5999b8a1f485c2ce98826fa5cd5199c28d4898c29a4e556"}
Oct 31 14:06:39.367006 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:39.366503    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v" event={"ID":"0dc7e2cf-7b12-4746-a474-705c25268e6a","Type":"ContainerStarted","Data":"87bce7b49795d8c1868f3a904a866eccb9e7d7398d905c9ddca3b7c86f59e1cb"}
Oct 31 14:06:39.386645 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:39.386591    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v" podStartSLOduration=2.38657522 podStartE2EDuration="2.38657522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:06:39.385210473 +0000 UTC m=+3796.783559111" watchObservedRunningTime="2026-10-31 14:06:39.38657522 +0000 UTC m=+3796.784923844"
Oct 31 14:06:40.376094 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:40.376056    4428 generic.go:356] "Generic (PLEG): container finished" podID="0dc7e2cf-7b12-4746-a474-705c25268e6a" containerID="6fd7483e8ecb12f8b5999b8a1f485c2ce98826fa5cd5199c28d4898c29a4e556" exitCode=0
Oct 31 14:06:40.376557 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:40.376169    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v" event={"ID":"0dc7e2cf-7b12-4746-a474-705c25268e6a","Type":"ContainerDied","Data":"6fd7483e8ecb12f8b5999b8a1f485c2ce98826fa5cd5199c28d4898c29a4e556"}
Oct 31 14:06:41.775074 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:41.774644    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"
Oct 31 14:06:41.911904 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:41.903199    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dc7e2cf-7b12-4746-a474-705c25268e6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc7e2cf-7b12-4746-a474-705c25268e6a-maas-api-xks-ca\") pod \"0dc7e2cf-7b12-4746-a474-705c25268e6a\" (UID: \"0dc7e2cf-7b12-4746-a474-705c25268e6a\") "
Oct 31 14:06:41.911904 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:41.903296    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dc7e2cf-7b12-4746-a474-705c25268e6a-kube-api-access-nbbjt\" (UniqueName: \"kubernetes.io/projected/0dc7e2cf-7b12-4746-a474-705c25268e6a-kube-api-access-nbbjt\") pod \"0dc7e2cf-7b12-4746-a474-705c25268e6a\" (UID: \"0dc7e2cf-7b12-4746-a474-705c25268e6a\") "
Oct 31 14:06:41.911904 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:41.903428    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dc7e2cf-7b12-4746-a474-705c25268e6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc7e2cf-7b12-4746-a474-705c25268e6a-maas-api-openshift-ca\") pod \"0dc7e2cf-7b12-4746-a474-705c25268e6a\" (UID: \"0dc7e2cf-7b12-4746-a474-705c25268e6a\") "
Oct 31 14:06:41.927909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:41.912460    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dc7e2cf-7b12-4746-a474-705c25268e6a-maas-api-openshift-ca" pod "0dc7e2cf-7b12-4746-a474-705c25268e6a" (UID: "0dc7e2cf-7b12-4746-a474-705c25268e6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:06:41.928369 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:41.928336    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dc7e2cf-7b12-4746-a474-705c25268e6a-maas-api-xks-ca" pod "0dc7e2cf-7b12-4746-a474-705c25268e6a" (UID: "0dc7e2cf-7b12-4746-a474-705c25268e6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:06:41.929739 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:41.929711    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc7e2cf-7b12-4746-a474-705c25268e6a-kube-api-access-nbbjt" pod "0dc7e2cf-7b12-4746-a474-705c25268e6a" (UID: "0dc7e2cf-7b12-4746-a474-705c25268e6a"). InnerVolumeSpecName "kube-api-access-nbbjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:06:42.005738 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:42.005570    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc7e2cf-7b12-4746-a474-705c25268e6a-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:06:42.005738 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:42.005621    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbbjt\" (UniqueName: \"kubernetes.io/projected/0dc7e2cf-7b12-4746-a474-705c25268e6a-kube-api-access-nbbjt\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:06:42.005738 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:42.005642    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc7e2cf-7b12-4746-a474-705c25268e6a-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:06:42.418986 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:42.418478    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v" event={"ID":"0dc7e2cf-7b12-4746-a474-705c25268e6a","Type":"ContainerDied","Data":"87bce7b49795d8c1868f3a904a866eccb9e7d7398d905c9ddca3b7c86f59e1cb"}
Oct 31 14:06:42.418986 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:42.418545    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87bce7b49795d8c1868f3a904a866eccb9e7d7398d905c9ddca3b7c86f59e1cb"
Oct 31 14:06:42.418986 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:42.418617    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"
Oct 31 14:06:44.000539 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:44.000500    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:06:48.738991 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:06:48.738951    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:06:52.040595 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:52.040544    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"]
Oct 31 14:06:52.045804 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:52.045771    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2580d87c-kwqpz"]
Oct 31 14:06:52.803208 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:52.803169    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b7e18df-36cf-4197-9a90-a2ff9fddc090" path="/var/lib/kubelet/pods/9b7e18df-36cf-4197-9a90-a2ff9fddc090/volumes"
Oct 31 14:06:53.027792 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:53.027748    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"]
Oct 31 14:06:53.031731 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:53.031699    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db2b73a6-tth5n"]
Oct 31 14:06:54.753367 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:06:54.753333    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2" path="/var/lib/kubelet/pods/f3f2ecf3-7c15-4d7f-b33f-75ce662d16c2/volumes"
Oct 31 14:07:04.374768 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.374677    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"]
Oct 31 14:07:04.375223 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.375174    4428 cpu_manager.go:414] "containerMap: removing container" podUID="0dc7e2cf-7b12-4746-a474-705c25268e6a" containerName="revoke-keys"
Oct 31 14:07:04.375223 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.375190    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc7e2cf-7b12-4746-a474-705c25268e6a" containerName="revoke-keys"
Oct 31 14:07:04.375305 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.375261    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc7e2cf-7b12-4746-a474-705c25268e6a" containerName="revoke-keys"
Oct 31 14:07:04.378655 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.378636    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"
Oct 31 14:07:04.388332 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.388300    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"]
Oct 31 14:07:04.389337 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.389315    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:07:04.389478 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.389359    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:07:04.477961 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.477909    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-42zxk\" (UID: \"43f7c0a1-2dee-4b55-8dcd-1cba4a981839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"
Oct 31 14:07:04.477961 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.477955    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-42zxk\" (UID: \"43f7c0a1-2dee-4b55-8dcd-1cba4a981839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"
Oct 31 14:07:04.478253 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.478135    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vvj5\" (UniqueName: \"kubernetes.io/projected/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-kube-api-access-4vvj5\") pod \"maas-api-revoke-sub-046e1343-42zxk\" (UID: \"43f7c0a1-2dee-4b55-8dcd-1cba4a981839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"
Oct 31 14:07:04.579098 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.579062    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vvj5\" (UniqueName: \"kubernetes.io/projected/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-kube-api-access-4vvj5\") pod \"maas-api-revoke-sub-046e1343-42zxk\" (UID: \"43f7c0a1-2dee-4b55-8dcd-1cba4a981839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"
Oct 31 14:07:04.579325 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.579133    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-42zxk\" (UID: \"43f7c0a1-2dee-4b55-8dcd-1cba4a981839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"
Oct 31 14:07:04.579325 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.579167    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-42zxk\" (UID: \"43f7c0a1-2dee-4b55-8dcd-1cba4a981839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"
Oct 31 14:07:04.579905 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.579860    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-42zxk\" (UID: \"43f7c0a1-2dee-4b55-8dcd-1cba4a981839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"
Oct 31 14:07:04.581747 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.581726    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-42zxk\" (UID: \"43f7c0a1-2dee-4b55-8dcd-1cba4a981839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"
Oct 31 14:07:04.593096 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.593069    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vvj5\" (UniqueName: \"kubernetes.io/projected/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-kube-api-access-4vvj5\") pod \"maas-api-revoke-sub-046e1343-42zxk\" (UID: \"43f7c0a1-2dee-4b55-8dcd-1cba4a981839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"
Oct 31 14:07:04.693833 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.693738    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"
Oct 31 14:07:04.981948 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:04.981857    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"]
Oct 31 14:07:04.990234 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:07:04.990204    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f7c0a1_2dee_4b55_8dcd_1cba4a981839.slice/crio-21a8ee68336b9fbadd6ec63d76215f2e3ad2c540098e9588a43037094b7aa248 WatchSource:0}: Error finding container 21a8ee68336b9fbadd6ec63d76215f2e3ad2c540098e9588a43037094b7aa248: Status 404 returned error can't find the container with id 21a8ee68336b9fbadd6ec63d76215f2e3ad2c540098e9588a43037094b7aa248
Oct 31 14:07:05.586391 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:05.586353    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk" event={"ID":"43f7c0a1-2dee-4b55-8dcd-1cba4a981839","Type":"ContainerStarted","Data":"18e0ad522119af8861bc16a5241525233ea7210cd69b3280100a2f746c4295bb"}
Oct 31 14:07:05.586949 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:05.586416    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk" event={"ID":"43f7c0a1-2dee-4b55-8dcd-1cba4a981839","Type":"ContainerStarted","Data":"21a8ee68336b9fbadd6ec63d76215f2e3ad2c540098e9588a43037094b7aa248"}
Oct 31 14:07:05.606432 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:05.606376    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk" podStartSLOduration=1.606360016 podStartE2EDuration="1.606360016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:07:05.606218485 +0000 UTC m=+3823.004567142" watchObservedRunningTime="2026-10-31 14:07:05.606360016 +0000 UTC m=+3823.004708639"
Oct 31 14:07:06.593245 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:06.593206    4428 generic.go:356] "Generic (PLEG): container finished" podID="43f7c0a1-2dee-4b55-8dcd-1cba4a981839" containerID="18e0ad522119af8861bc16a5241525233ea7210cd69b3280100a2f746c4295bb" exitCode=0
Oct 31 14:07:06.593740 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:06.593287    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk" event={"ID":"43f7c0a1-2dee-4b55-8dcd-1cba4a981839","Type":"ContainerDied","Data":"18e0ad522119af8861bc16a5241525233ea7210cd69b3280100a2f746c4295bb"}
Oct 31 14:07:07.733902 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:07.733854    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"
Oct 31 14:07:07.814854 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:07.814818    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-maas-api-openshift-ca\") pod \"43f7c0a1-2dee-4b55-8dcd-1cba4a981839\" (UID: \"43f7c0a1-2dee-4b55-8dcd-1cba4a981839\") "
Oct 31 14:07:07.815097 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:07.814894    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-maas-api-xks-ca\") pod \"43f7c0a1-2dee-4b55-8dcd-1cba4a981839\" (UID: \"43f7c0a1-2dee-4b55-8dcd-1cba4a981839\") "
Oct 31 14:07:07.815097 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:07.814927    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-kube-api-access-4vvj5\" (UniqueName: \"kubernetes.io/projected/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-kube-api-access-4vvj5\") pod \"43f7c0a1-2dee-4b55-8dcd-1cba4a981839\" (UID: \"43f7c0a1-2dee-4b55-8dcd-1cba4a981839\") "
Oct 31 14:07:07.815245 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:07.815194    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-maas-api-openshift-ca" pod "43f7c0a1-2dee-4b55-8dcd-1cba4a981839" (UID: "43f7c0a1-2dee-4b55-8dcd-1cba4a981839"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:07:07.815366 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:07.815346    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:07.817248 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:07.817218    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-maas-api-xks-ca" pod "43f7c0a1-2dee-4b55-8dcd-1cba4a981839" (UID: "43f7c0a1-2dee-4b55-8dcd-1cba4a981839"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:07:07.817356 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:07.817274    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-kube-api-access-4vvj5" pod "43f7c0a1-2dee-4b55-8dcd-1cba4a981839" (UID: "43f7c0a1-2dee-4b55-8dcd-1cba4a981839"). InnerVolumeSpecName "kube-api-access-4vvj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:07:07.917195 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:07.916982    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:07.917195 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:07.917022    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vvj5\" (UniqueName: \"kubernetes.io/projected/43f7c0a1-2dee-4b55-8dcd-1cba4a981839-kube-api-access-4vvj5\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:08.605535 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:08.605504    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"
Oct 31 14:07:08.605827 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:08.605538    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk" event={"ID":"43f7c0a1-2dee-4b55-8dcd-1cba4a981839","Type":"ContainerDied","Data":"21a8ee68336b9fbadd6ec63d76215f2e3ad2c540098e9588a43037094b7aa248"}
Oct 31 14:07:08.605827 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:08.605597    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21a8ee68336b9fbadd6ec63d76215f2e3ad2c540098e9588a43037094b7aa248"
Oct 31 14:07:14.605751 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.605712    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"]
Oct 31 14:07:14.606418 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.606401    4428 cpu_manager.go:414] "containerMap: removing container" podUID="43f7c0a1-2dee-4b55-8dcd-1cba4a981839" containerName="revoke-keys"
Oct 31 14:07:14.606512 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.606422    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43f7c0a1-2dee-4b55-8dcd-1cba4a981839" containerName="revoke-keys"
Oct 31 14:07:14.606570 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.606532    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="43f7c0a1-2dee-4b55-8dcd-1cba4a981839" containerName="revoke-keys"
Oct 31 14:07:14.609832 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.609807    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"
Oct 31 14:07:14.618423 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.618396    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:07:14.618581 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.618507    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:07:14.621693 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.621658    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"]
Oct 31 14:07:14.781922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.781828    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff36a20-8c96-4014-84a5-0f5be5a87e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8fjkk\" (UID: \"bff36a20-8c96-4014-84a5-0f5be5a87e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"
Oct 31 14:07:14.782120 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.782042    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff36a20-8c96-4014-84a5-0f5be5a87e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8fjkk\" (UID: \"bff36a20-8c96-4014-84a5-0f5be5a87e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"
Oct 31 14:07:14.782120 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.782080    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgkb7\" (UniqueName: \"kubernetes.io/projected/bff36a20-8c96-4014-84a5-0f5be5a87e53-kube-api-access-bgkb7\") pod \"maas-api-revoke-sub-d3fba056-8fjkk\" (UID: \"bff36a20-8c96-4014-84a5-0f5be5a87e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"
Oct 31 14:07:14.883243 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.883148    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff36a20-8c96-4014-84a5-0f5be5a87e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8fjkk\" (UID: \"bff36a20-8c96-4014-84a5-0f5be5a87e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"
Oct 31 14:07:14.883243 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.883193    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgkb7\" (UniqueName: \"kubernetes.io/projected/bff36a20-8c96-4014-84a5-0f5be5a87e53-kube-api-access-bgkb7\") pod \"maas-api-revoke-sub-d3fba056-8fjkk\" (UID: \"bff36a20-8c96-4014-84a5-0f5be5a87e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"
Oct 31 14:07:14.883243 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.883240    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff36a20-8c96-4014-84a5-0f5be5a87e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8fjkk\" (UID: \"bff36a20-8c96-4014-84a5-0f5be5a87e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"
Oct 31 14:07:14.884087 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.883962    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff36a20-8c96-4014-84a5-0f5be5a87e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8fjkk\" (UID: \"bff36a20-8c96-4014-84a5-0f5be5a87e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"
Oct 31 14:07:14.885735 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.885709    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff36a20-8c96-4014-84a5-0f5be5a87e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8fjkk\" (UID: \"bff36a20-8c96-4014-84a5-0f5be5a87e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"
Oct 31 14:07:14.896796 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.896772    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgkb7\" (UniqueName: \"kubernetes.io/projected/bff36a20-8c96-4014-84a5-0f5be5a87e53-kube-api-access-bgkb7\") pod \"maas-api-revoke-sub-d3fba056-8fjkk\" (UID: \"bff36a20-8c96-4014-84a5-0f5be5a87e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"
Oct 31 14:07:14.927105 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:14.927063    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"
Oct 31 14:07:15.207973 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:15.207938    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"]
Oct 31 14:07:15.210067 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:07:15.210038    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff36a20_8c96_4014_84a5_0f5be5a87e53.slice/crio-1da6add3a9a10d1675a87451944273b6b55862df2663f56450f98ed454a9a221 WatchSource:0}: Error finding container 1da6add3a9a10d1675a87451944273b6b55862df2663f56450f98ed454a9a221: Status 404 returned error can't find the container with id 1da6add3a9a10d1675a87451944273b6b55862df2663f56450f98ed454a9a221
Oct 31 14:07:15.648739 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:15.648691    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk" event={"ID":"bff36a20-8c96-4014-84a5-0f5be5a87e53","Type":"ContainerStarted","Data":"0bcd258fcf2804a89d13ff5ca15ee48f54469aec65a73971cde45d212c32564f"}
Oct 31 14:07:15.649193 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:15.648751    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk" event={"ID":"bff36a20-8c96-4014-84a5-0f5be5a87e53","Type":"ContainerStarted","Data":"1da6add3a9a10d1675a87451944273b6b55862df2663f56450f98ed454a9a221"}
Oct 31 14:07:15.678449 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:15.678385    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk" podStartSLOduration=1.6783650350000001 podStartE2EDuration="1.678365035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:07:15.676282008 +0000 UTC m=+3833.074630633" watchObservedRunningTime="2026-10-31 14:07:15.678365035 +0000 UTC m=+3833.076713652"
Oct 31 14:07:16.016130 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:16.016077    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:07:16.684820 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:16.684773    4428 generic.go:356] "Generic (PLEG): container finished" podID="bff36a20-8c96-4014-84a5-0f5be5a87e53" containerID="0bcd258fcf2804a89d13ff5ca15ee48f54469aec65a73971cde45d212c32564f" exitCode=0
Oct 31 14:07:16.685559 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:16.685527    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk" event={"ID":"bff36a20-8c96-4014-84a5-0f5be5a87e53","Type":"ContainerDied","Data":"0bcd258fcf2804a89d13ff5ca15ee48f54469aec65a73971cde45d212c32564f"}
Oct 31 14:07:16.745129 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:16.745093    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:07:18.145141 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:18.145112    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"
Oct 31 14:07:18.240423 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:18.240388    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff36a20-8c96-4014-84a5-0f5be5a87e53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff36a20-8c96-4014-84a5-0f5be5a87e53-maas-api-openshift-ca\") pod \"bff36a20-8c96-4014-84a5-0f5be5a87e53\" (UID: \"bff36a20-8c96-4014-84a5-0f5be5a87e53\") "
Oct 31 14:07:18.240647 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:18.240548    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff36a20-8c96-4014-84a5-0f5be5a87e53-kube-api-access-bgkb7\" (UniqueName: \"kubernetes.io/projected/bff36a20-8c96-4014-84a5-0f5be5a87e53-kube-api-access-bgkb7\") pod \"bff36a20-8c96-4014-84a5-0f5be5a87e53\" (UID: \"bff36a20-8c96-4014-84a5-0f5be5a87e53\") "
Oct 31 14:07:18.240647 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:18.240625    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bff36a20-8c96-4014-84a5-0f5be5a87e53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff36a20-8c96-4014-84a5-0f5be5a87e53-maas-api-xks-ca\") pod \"bff36a20-8c96-4014-84a5-0f5be5a87e53\" (UID: \"bff36a20-8c96-4014-84a5-0f5be5a87e53\") "
Oct 31 14:07:18.240924 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:18.240862    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff36a20-8c96-4014-84a5-0f5be5a87e53-maas-api-openshift-ca" pod "bff36a20-8c96-4014-84a5-0f5be5a87e53" (UID: "bff36a20-8c96-4014-84a5-0f5be5a87e53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:07:18.243074 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:18.243034    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff36a20-8c96-4014-84a5-0f5be5a87e53-kube-api-access-bgkb7" pod "bff36a20-8c96-4014-84a5-0f5be5a87e53" (UID: "bff36a20-8c96-4014-84a5-0f5be5a87e53"). InnerVolumeSpecName "kube-api-access-bgkb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:07:18.243207 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:18.243131    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff36a20-8c96-4014-84a5-0f5be5a87e53-maas-api-xks-ca" pod "bff36a20-8c96-4014-84a5-0f5be5a87e53" (UID: "bff36a20-8c96-4014-84a5-0f5be5a87e53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:07:18.341904 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:18.341812    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgkb7\" (UniqueName: \"kubernetes.io/projected/bff36a20-8c96-4014-84a5-0f5be5a87e53-kube-api-access-bgkb7\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:18.341904 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:18.341902    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff36a20-8c96-4014-84a5-0f5be5a87e53-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:18.341904 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:18.341918    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff36a20-8c96-4014-84a5-0f5be5a87e53-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:18.703721 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:18.703687    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"
Oct 31 14:07:18.703980 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:18.703685    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk" event={"ID":"bff36a20-8c96-4014-84a5-0f5be5a87e53","Type":"ContainerDied","Data":"1da6add3a9a10d1675a87451944273b6b55862df2663f56450f98ed454a9a221"}
Oct 31 14:07:18.703980 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:18.703841    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1da6add3a9a10d1675a87451944273b6b55862df2663f56450f98ed454a9a221"
Oct 31 14:07:18.740136 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:07:18.740086    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:07:25.027270 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:25.027235    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"]
Oct 31 14:07:25.032620 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:25.032591    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baaaafb4-nfkrp"]
Oct 31 14:07:26.754229 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:26.754194    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6" path="/var/lib/kubelet/pods/4dfd0f1c-8e15-4d6c-bdef-34581cbe28d6/volumes"
Oct 31 14:07:35.493127 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.493085    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"]
Oct 31 14:07:35.493556 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.493515    4428 cpu_manager.go:414] "containerMap: removing container" podUID="bff36a20-8c96-4014-84a5-0f5be5a87e53" containerName="revoke-keys"
Oct 31 14:07:35.493556 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.493528    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff36a20-8c96-4014-84a5-0f5be5a87e53" containerName="revoke-keys"
Oct 31 14:07:35.493634 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.493607    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff36a20-8c96-4014-84a5-0f5be5a87e53" containerName="revoke-keys"
Oct 31 14:07:35.496813 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.496797    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"
Oct 31 14:07:35.503597 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.503564    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:07:35.505376 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.505090    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:07:35.506024 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.505996    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"]
Oct 31 14:07:35.644989 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.644951    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff96a69f-cdae-4bb0-9575-a5a685a300e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4xnz4\" (UID: \"ff96a69f-cdae-4bb0-9575-a5a685a300e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"
Oct 31 14:07:35.645187 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.645068    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbbqq\" (UniqueName: \"kubernetes.io/projected/ff96a69f-cdae-4bb0-9575-a5a685a300e6-kube-api-access-mbbqq\") pod \"maas-api-revoke-sub-ae594979-4xnz4\" (UID: \"ff96a69f-cdae-4bb0-9575-a5a685a300e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"
Oct 31 14:07:35.645187 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.645122    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff96a69f-cdae-4bb0-9575-a5a685a300e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4xnz4\" (UID: \"ff96a69f-cdae-4bb0-9575-a5a685a300e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"
Oct 31 14:07:35.746177 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.746083    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbbqq\" (UniqueName: \"kubernetes.io/projected/ff96a69f-cdae-4bb0-9575-a5a685a300e6-kube-api-access-mbbqq\") pod \"maas-api-revoke-sub-ae594979-4xnz4\" (UID: \"ff96a69f-cdae-4bb0-9575-a5a685a300e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"
Oct 31 14:07:35.746177 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.746130    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff96a69f-cdae-4bb0-9575-a5a685a300e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4xnz4\" (UID: \"ff96a69f-cdae-4bb0-9575-a5a685a300e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"
Oct 31 14:07:35.746177 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.746177    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff96a69f-cdae-4bb0-9575-a5a685a300e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4xnz4\" (UID: \"ff96a69f-cdae-4bb0-9575-a5a685a300e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"
Oct 31 14:07:35.746813 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.746791    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff96a69f-cdae-4bb0-9575-a5a685a300e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4xnz4\" (UID: \"ff96a69f-cdae-4bb0-9575-a5a685a300e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"
Oct 31 14:07:35.748787 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.748766    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff96a69f-cdae-4bb0-9575-a5a685a300e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4xnz4\" (UID: \"ff96a69f-cdae-4bb0-9575-a5a685a300e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"
Oct 31 14:07:35.760946 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.760915    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbbqq\" (UniqueName: \"kubernetes.io/projected/ff96a69f-cdae-4bb0-9575-a5a685a300e6-kube-api-access-mbbqq\") pod \"maas-api-revoke-sub-ae594979-4xnz4\" (UID: \"ff96a69f-cdae-4bb0-9575-a5a685a300e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"
Oct 31 14:07:35.812669 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.812622    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"
Oct 31 14:07:35.965068 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.965034    4428 scope.go:122] "RemoveContainer" containerID="a740dc3d9beabaea02feb2ce7f21dc4c90e229b4fedde717fdd8d21d49aa7fa0"
Oct 31 14:07:35.974720 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.974699    4428 scope.go:122] "RemoveContainer" containerID="0cedc121da47030aea7e014337fd8dcc61b3edde3a694addc9a956d5d94741f9"
Oct 31 14:07:35.983630 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:35.983608    4428 scope.go:122] "RemoveContainer" containerID="d9eea432b31b7e7ed40d0b83026141bf68772cd61acb5e2c6e9ebea6bcc18975"
Oct 31 14:07:36.105284 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:36.105254    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"]
Oct 31 14:07:36.106682 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:07:36.106645    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff96a69f_cdae_4bb0_9575_a5a685a300e6.slice/crio-1dcf95f2acd256ac07e5607097e8ad0aaa4c3a563b94b0510f1b8fc04eed428d WatchSource:0}: Error finding container 1dcf95f2acd256ac07e5607097e8ad0aaa4c3a563b94b0510f1b8fc04eed428d: Status 404 returned error can't find the container with id 1dcf95f2acd256ac07e5607097e8ad0aaa4c3a563b94b0510f1b8fc04eed428d
Oct 31 14:07:36.861683 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:36.861641    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4" event={"ID":"ff96a69f-cdae-4bb0-9575-a5a685a300e6","Type":"ContainerStarted","Data":"c7c3afc7115c09af269e3be6b4fa2d7502c2c6f1b121b8d49ad03fc08560a4b0"}
Oct 31 14:07:36.862141 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:36.861704    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4" event={"ID":"ff96a69f-cdae-4bb0-9575-a5a685a300e6","Type":"ContainerStarted","Data":"1dcf95f2acd256ac07e5607097e8ad0aaa4c3a563b94b0510f1b8fc04eed428d"}
Oct 31 14:07:36.883741 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:36.883684    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4" podStartSLOduration=1.883669319 podStartE2EDuration="1.883669319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:07:36.882084163 +0000 UTC m=+3854.280432792" watchObservedRunningTime="2026-10-31 14:07:36.883669319 +0000 UTC m=+3854.282017944"
Oct 31 14:07:37.869224 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:37.869180    4428 generic.go:356] "Generic (PLEG): container finished" podID="ff96a69f-cdae-4bb0-9575-a5a685a300e6" containerID="c7c3afc7115c09af269e3be6b4fa2d7502c2c6f1b121b8d49ad03fc08560a4b0" exitCode=0
Oct 31 14:07:37.869708 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:37.869213    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4" event={"ID":"ff96a69f-cdae-4bb0-9575-a5a685a300e6","Type":"ContainerDied","Data":"c7c3afc7115c09af269e3be6b4fa2d7502c2c6f1b121b8d49ad03fc08560a4b0"}
Oct 31 14:07:39.022529 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:39.022498    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"
Oct 31 14:07:39.179703 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:39.179615    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff96a69f-cdae-4bb0-9575-a5a685a300e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff96a69f-cdae-4bb0-9575-a5a685a300e6-maas-api-xks-ca\") pod \"ff96a69f-cdae-4bb0-9575-a5a685a300e6\" (UID: \"ff96a69f-cdae-4bb0-9575-a5a685a300e6\") "
Oct 31 14:07:39.179910 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:39.179716    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff96a69f-cdae-4bb0-9575-a5a685a300e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff96a69f-cdae-4bb0-9575-a5a685a300e6-maas-api-openshift-ca\") pod \"ff96a69f-cdae-4bb0-9575-a5a685a300e6\" (UID: \"ff96a69f-cdae-4bb0-9575-a5a685a300e6\") "
Oct 31 14:07:39.179910 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:39.179829    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff96a69f-cdae-4bb0-9575-a5a685a300e6-kube-api-access-mbbqq\" (UniqueName: \"kubernetes.io/projected/ff96a69f-cdae-4bb0-9575-a5a685a300e6-kube-api-access-mbbqq\") pod \"ff96a69f-cdae-4bb0-9575-a5a685a300e6\" (UID: \"ff96a69f-cdae-4bb0-9575-a5a685a300e6\") "
Oct 31 14:07:39.180165 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:39.180129    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff96a69f-cdae-4bb0-9575-a5a685a300e6-maas-api-openshift-ca" pod "ff96a69f-cdae-4bb0-9575-a5a685a300e6" (UID: "ff96a69f-cdae-4bb0-9575-a5a685a300e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:07:39.182019 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:39.181992    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff96a69f-cdae-4bb0-9575-a5a685a300e6-maas-api-xks-ca" pod "ff96a69f-cdae-4bb0-9575-a5a685a300e6" (UID: "ff96a69f-cdae-4bb0-9575-a5a685a300e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:07:39.182273 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:39.182254    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff96a69f-cdae-4bb0-9575-a5a685a300e6-kube-api-access-mbbqq" pod "ff96a69f-cdae-4bb0-9575-a5a685a300e6" (UID: "ff96a69f-cdae-4bb0-9575-a5a685a300e6"). InnerVolumeSpecName "kube-api-access-mbbqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:07:39.280598 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:39.280555    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff96a69f-cdae-4bb0-9575-a5a685a300e6-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:39.280598 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:39.280589    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbbqq\" (UniqueName: \"kubernetes.io/projected/ff96a69f-cdae-4bb0-9575-a5a685a300e6-kube-api-access-mbbqq\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:39.280598 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:39.280599    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff96a69f-cdae-4bb0-9575-a5a685a300e6-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:39.881589 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:39.881552    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4" event={"ID":"ff96a69f-cdae-4bb0-9575-a5a685a300e6","Type":"ContainerDied","Data":"1dcf95f2acd256ac07e5607097e8ad0aaa4c3a563b94b0510f1b8fc04eed428d"}
Oct 31 14:07:39.881589 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:39.881580    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"
Oct 31 14:07:39.881822 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:39.881602    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dcf95f2acd256ac07e5607097e8ad0aaa4c3a563b94b0510f1b8fc04eed428d"
Oct 31 14:07:41.039415 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:41.039380    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"]
Oct 31 14:07:41.042110 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:41.042062    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"]
Oct 31 14:07:41.044199 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:41.044175    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3bb47f6-qvsvt"]
Oct 31 14:07:41.046131 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:41.046105    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5028dced-8wprp"]
Oct 31 14:07:42.754104 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:42.754066    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1369bea-d9f3-4182-8a23-ee5a0a7c90a5" path="/var/lib/kubelet/pods/a1369bea-d9f3-4182-8a23-ee5a0a7c90a5/volumes"
Oct 31 14:07:42.754688 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:42.754667    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b24885cc-c332-444e-89ba-9232a2b43196" path="/var/lib/kubelet/pods/b24885cc-c332-444e-89ba-9232a2b43196/volumes"
Oct 31 14:07:48.745049 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:07:48.744974    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:07:52.029241 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:52.029197    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"]
Oct 31 14:07:52.034878 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:52.034837    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb33258-s49k8"]
Oct 31 14:07:52.754236 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:52.754192    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a6debd-03b9-4487-9d39-b9fa19078ea9" path="/var/lib/kubelet/pods/07a6debd-03b9-4487-9d39-b9fa19078ea9/volumes"
Oct 31 14:07:53.724635 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:53.724593    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"]
Oct 31 14:07:53.725182 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:53.725162    4428 cpu_manager.go:414] "containerMap: removing container" podUID="ff96a69f-cdae-4bb0-9575-a5a685a300e6" containerName="revoke-keys"
Oct 31 14:07:53.725230 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:53.725185    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff96a69f-cdae-4bb0-9575-a5a685a300e6" containerName="revoke-keys"
Oct 31 14:07:53.725280 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:53.725256    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff96a69f-cdae-4bb0-9575-a5a685a300e6" containerName="revoke-keys"
Oct 31 14:07:53.729904 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:53.729862    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"
Oct 31 14:07:53.738179 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:53.738150    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:07:53.738318 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:53.738208    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:07:53.741080 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:53.741055    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"]
Oct 31 14:07:53.744122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:53.744089    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5348bb10-c028-49c7-9c11-6f90f6f1e63d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-n7nf6\" (UID: \"5348bb10-c028-49c7-9c11-6f90f6f1e63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"
Oct 31 14:07:53.744268 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:53.744244    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-768fp\" (UniqueName: \"kubernetes.io/projected/5348bb10-c028-49c7-9c11-6f90f6f1e63d-kube-api-access-768fp\") pod \"maas-api-revoke-sub-a4aef7d7-n7nf6\" (UID: \"5348bb10-c028-49c7-9c11-6f90f6f1e63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"
Oct 31 14:07:53.744342 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:53.744289    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5348bb10-c028-49c7-9c11-6f90f6f1e63d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-n7nf6\" (UID: \"5348bb10-c028-49c7-9c11-6f90f6f1e63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"
Oct 31 14:07:53.845434 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:53.845388    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-768fp\" (UniqueName: \"kubernetes.io/projected/5348bb10-c028-49c7-9c11-6f90f6f1e63d-kube-api-access-768fp\") pod \"maas-api-revoke-sub-a4aef7d7-n7nf6\" (UID: \"5348bb10-c028-49c7-9c11-6f90f6f1e63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"
Oct 31 14:07:53.845434 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:53.845437    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5348bb10-c028-49c7-9c11-6f90f6f1e63d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-n7nf6\" (UID: \"5348bb10-c028-49c7-9c11-6f90f6f1e63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"
Oct 31 14:07:53.845724 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:53.845488    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5348bb10-c028-49c7-9c11-6f90f6f1e63d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-n7nf6\" (UID: \"5348bb10-c028-49c7-9c11-6f90f6f1e63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"
Oct 31 14:07:53.846312 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:53.846289    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5348bb10-c028-49c7-9c11-6f90f6f1e63d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-n7nf6\" (UID: \"5348bb10-c028-49c7-9c11-6f90f6f1e63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"
Oct 31 14:07:53.848088 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:53.848066    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5348bb10-c028-49c7-9c11-6f90f6f1e63d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-n7nf6\" (UID: \"5348bb10-c028-49c7-9c11-6f90f6f1e63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"
Oct 31 14:07:53.859038 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:53.859009    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-768fp\" (UniqueName: \"kubernetes.io/projected/5348bb10-c028-49c7-9c11-6f90f6f1e63d-kube-api-access-768fp\") pod \"maas-api-revoke-sub-a4aef7d7-n7nf6\" (UID: \"5348bb10-c028-49c7-9c11-6f90f6f1e63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"
Oct 31 14:07:54.045335 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:54.045238    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"
Oct 31 14:07:54.326353 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:54.326320    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"]
Oct 31 14:07:54.330302 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:07:54.330254    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5348bb10_c028_49c7_9c11_6f90f6f1e63d.slice/crio-d4ab21e508363de706cf282e2f314151dac4e2be78da1fa1ecff6d6ee4eac025 WatchSource:0}: Error finding container d4ab21e508363de706cf282e2f314151dac4e2be78da1fa1ecff6d6ee4eac025: Status 404 returned error can't find the container with id d4ab21e508363de706cf282e2f314151dac4e2be78da1fa1ecff6d6ee4eac025
Oct 31 14:07:54.968722 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:54.968681    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6" event={"ID":"5348bb10-c028-49c7-9c11-6f90f6f1e63d","Type":"ContainerStarted","Data":"fbafaf1d0ad90475336ed6ea502857b9ea1adeb48128feaf216cfc25aefdeabe"}
Oct 31 14:07:54.969145 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:54.968744    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6" event={"ID":"5348bb10-c028-49c7-9c11-6f90f6f1e63d","Type":"ContainerStarted","Data":"d4ab21e508363de706cf282e2f314151dac4e2be78da1fa1ecff6d6ee4eac025"}
Oct 31 14:07:54.991609 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:54.991556    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6" podStartSLOduration=1.9915410869999999 podStartE2EDuration="1.991541087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:07:54.989133223 +0000 UTC m=+3872.387481848" watchObservedRunningTime="2026-10-31 14:07:54.991541087 +0000 UTC m=+3872.389889711"
Oct 31 14:07:55.001219 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.001188    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"]
Oct 31 14:07:55.001396 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.001378    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx" podUID="2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92" containerName="discovery" containerID="cri-o://43408c2e16b08f45fac843b0024d32abd4b45825f7b2639aa7f5bcb6699ca717" gracePeriod=30
Oct 31 14:07:55.334076 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.334048    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 14:07:55.358575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.358250    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-csr-dns-cert\") pod \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") "
Oct 31 14:07:55.358575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.358303    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-kubeconfig\") pod \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") "
Oct 31 14:07:55.358575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.358351    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-token\" (UniqueName: \"kubernetes.io/projected/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-token\") pod \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") "
Oct 31 14:07:55.358575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.358389    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-local-certs\") pod \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") "
Oct 31 14:07:55.358989 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.358581    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-kube-api-access-48fhq\" (UniqueName: \"kubernetes.io/projected/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-kube-api-access-48fhq\") pod \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") "
Oct 31 14:07:55.358989 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.358657    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-csr-ca-configmap\") pod \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") "
Oct 31 14:07:55.358989 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.358730    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-cacerts\" (UniqueName: \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-cacerts\") pod \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\" (UID: \"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92\") "
Oct 31 14:07:55.359903 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.359611    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-csr-ca-configmap" pod "2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92" (UID: "2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:07:55.359903 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.359836    4428 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-csr-ca-configmap\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:55.361850 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.361818    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-local-certs" pod "2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92" (UID: "2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:07:55.362911 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.362834    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-kubeconfig" pod "2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92" (UID: "2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:07:55.362911 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.362902    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-csr-dns-cert" pod "2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92" (UID: "2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:07:55.363140 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.363116    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-cacerts" pod "2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92" (UID: "2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:07:55.363329 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.363301    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-kube-api-access-48fhq" pod "2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92" (UID: "2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92"). InnerVolumeSpecName "kube-api-access-48fhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:07:55.364096 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.364076    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-token" pod "2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92" (UID: "2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:07:55.460894 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.460815    4428 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-local-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:55.460894 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.460887    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48fhq\" (UniqueName: \"kubernetes.io/projected/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-kube-api-access-48fhq\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:55.461132 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.460906    4428 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-cacerts\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:55.461132 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.460921    4428 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-csr-dns-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:55.461132 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.460936    4428 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-kubeconfig\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:55.461132 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.460949    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:55.976038 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.976002    4428 generic.go:356] "Generic (PLEG): container finished" podID="2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92" containerID="43408c2e16b08f45fac843b0024d32abd4b45825f7b2639aa7f5bcb6699ca717" exitCode=0
Oct 31 14:07:55.976641 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.976083    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"
Oct 31 14:07:55.976641 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.976097    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx" event={"ID":"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92","Type":"ContainerDied","Data":"43408c2e16b08f45fac843b0024d32abd4b45825f7b2639aa7f5bcb6699ca717"}
Oct 31 14:07:55.976641 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.976164    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx" event={"ID":"2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92","Type":"ContainerDied","Data":"284454f332aaf6daecf7b90d5d0f3fff8a1a1d869e7f78f037e5e50f656a9cae"}
Oct 31 14:07:55.976641 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.976191    4428 scope.go:122] "RemoveContainer" containerID="43408c2e16b08f45fac843b0024d32abd4b45825f7b2639aa7f5bcb6699ca717"
Oct 31 14:07:55.977856 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.977828    4428 generic.go:356] "Generic (PLEG): container finished" podID="5348bb10-c028-49c7-9c11-6f90f6f1e63d" containerID="fbafaf1d0ad90475336ed6ea502857b9ea1adeb48128feaf216cfc25aefdeabe" exitCode=0
Oct 31 14:07:55.978015 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.977897    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6" event={"ID":"5348bb10-c028-49c7-9c11-6f90f6f1e63d","Type":"ContainerDied","Data":"fbafaf1d0ad90475336ed6ea502857b9ea1adeb48128feaf216cfc25aefdeabe"}
Oct 31 14:07:55.997990 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.997969    4428 scope.go:122] "RemoveContainer" containerID="43408c2e16b08f45fac843b0024d32abd4b45825f7b2639aa7f5bcb6699ca717"
Oct 31 14:07:55.998298 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:07:55.998274    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43408c2e16b08f45fac843b0024d32abd4b45825f7b2639aa7f5bcb6699ca717\": container with ID starting with 43408c2e16b08f45fac843b0024d32abd4b45825f7b2639aa7f5bcb6699ca717 not found: ID does not exist" containerID="43408c2e16b08f45fac843b0024d32abd4b45825f7b2639aa7f5bcb6699ca717"
Oct 31 14:07:55.998377 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:55.998307    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43408c2e16b08f45fac843b0024d32abd4b45825f7b2639aa7f5bcb6699ca717"} err="failed to get container status \"43408c2e16b08f45fac843b0024d32abd4b45825f7b2639aa7f5bcb6699ca717\": rpc error: code = NotFound desc = could not find container \"43408c2e16b08f45fac843b0024d32abd4b45825f7b2639aa7f5bcb6699ca717\": container with ID starting with 43408c2e16b08f45fac843b0024d32abd4b45825f7b2639aa7f5bcb6699ca717 not found: ID does not exist"
Oct 31 14:07:56.017914 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:56.017857    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"]
Oct 31 14:07:56.023845 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:56.023813    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-thzmx"]
Oct 31 14:07:56.754667 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:56.754634    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92" path="/var/lib/kubelet/pods/2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92/volumes"
Oct 31 14:07:57.117653 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:57.117622    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"
Oct 31 14:07:57.175815 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:57.175782    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5348bb10-c028-49c7-9c11-6f90f6f1e63d-kube-api-access-768fp\" (UniqueName: \"kubernetes.io/projected/5348bb10-c028-49c7-9c11-6f90f6f1e63d-kube-api-access-768fp\") pod \"5348bb10-c028-49c7-9c11-6f90f6f1e63d\" (UID: \"5348bb10-c028-49c7-9c11-6f90f6f1e63d\") "
Oct 31 14:07:57.175998 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:57.175833    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5348bb10-c028-49c7-9c11-6f90f6f1e63d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5348bb10-c028-49c7-9c11-6f90f6f1e63d-maas-api-xks-ca\") pod \"5348bb10-c028-49c7-9c11-6f90f6f1e63d\" (UID: \"5348bb10-c028-49c7-9c11-6f90f6f1e63d\") "
Oct 31 14:07:57.175998 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:57.175938    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5348bb10-c028-49c7-9c11-6f90f6f1e63d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5348bb10-c028-49c7-9c11-6f90f6f1e63d-maas-api-openshift-ca\") pod \"5348bb10-c028-49c7-9c11-6f90f6f1e63d\" (UID: \"5348bb10-c028-49c7-9c11-6f90f6f1e63d\") "
Oct 31 14:07:57.176352 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:57.176328    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5348bb10-c028-49c7-9c11-6f90f6f1e63d-maas-api-openshift-ca" pod "5348bb10-c028-49c7-9c11-6f90f6f1e63d" (UID: "5348bb10-c028-49c7-9c11-6f90f6f1e63d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:07:57.178290 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:57.178270    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5348bb10-c028-49c7-9c11-6f90f6f1e63d-maas-api-xks-ca" pod "5348bb10-c028-49c7-9c11-6f90f6f1e63d" (UID: "5348bb10-c028-49c7-9c11-6f90f6f1e63d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:07:57.178290 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:57.178271    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5348bb10-c028-49c7-9c11-6f90f6f1e63d-kube-api-access-768fp" pod "5348bb10-c028-49c7-9c11-6f90f6f1e63d" (UID: "5348bb10-c028-49c7-9c11-6f90f6f1e63d"). InnerVolumeSpecName "kube-api-access-768fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:07:57.277076 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:57.277042    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5348bb10-c028-49c7-9c11-6f90f6f1e63d-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:57.277076 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:57.277072    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-768fp\" (UniqueName: \"kubernetes.io/projected/5348bb10-c028-49c7-9c11-6f90f6f1e63d-kube-api-access-768fp\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:57.277076 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:57.277083    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5348bb10-c028-49c7-9c11-6f90f6f1e63d-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:07:57.992238 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:57.992209    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"
Oct 31 14:07:57.992430 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:57.992204    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6" event={"ID":"5348bb10-c028-49c7-9c11-6f90f6f1e63d","Type":"ContainerDied","Data":"d4ab21e508363de706cf282e2f314151dac4e2be78da1fa1ecff6d6ee4eac025"}
Oct 31 14:07:57.992430 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:07:57.992342    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4ab21e508363de706cf282e2f314151dac4e2be78da1fa1ecff6d6ee4eac025"
Oct 31 14:08:12.643757 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.643721    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"]
Oct 31 14:08:12.644276 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.644161    4428 cpu_manager.go:414] "containerMap: removing container" podUID="5348bb10-c028-49c7-9c11-6f90f6f1e63d" containerName="revoke-keys"
Oct 31 14:08:12.644276 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.644175    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5348bb10-c028-49c7-9c11-6f90f6f1e63d" containerName="revoke-keys"
Oct 31 14:08:12.644276 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.644209    4428 cpu_manager.go:414] "containerMap: removing container" podUID="2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92" containerName="discovery"
Oct 31 14:08:12.644276 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.644214    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92" containerName="discovery"
Oct 31 14:08:12.644276 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.644277    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eda1ce0-cc32-4f11-8bd2-c3fe7a42cb92" containerName="discovery"
Oct 31 14:08:12.644473 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.644290    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5348bb10-c028-49c7-9c11-6f90f6f1e63d" containerName="revoke-keys"
Oct 31 14:08:12.647616 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.647587    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"
Oct 31 14:08:12.661672 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.661640    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:08:12.661832 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.661686    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:08:12.662705 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.662674    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"]
Oct 31 14:08:12.743195 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.743157    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc8sz\" (UniqueName: \"kubernetes.io/projected/5899d096-c8b6-4b3c-979c-45e981e5acc8-kube-api-access-tc8sz\") pod \"maas-api-revoke-sub-79805ca9-9zkb7\" (UID: \"5899d096-c8b6-4b3c-979c-45e981e5acc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"
Oct 31 14:08:12.743383 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.743200    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5899d096-c8b6-4b3c-979c-45e981e5acc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9zkb7\" (UID: \"5899d096-c8b6-4b3c-979c-45e981e5acc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"
Oct 31 14:08:12.743383 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.743287    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5899d096-c8b6-4b3c-979c-45e981e5acc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9zkb7\" (UID: \"5899d096-c8b6-4b3c-979c-45e981e5acc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"
Oct 31 14:08:12.844115 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.844058    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc8sz\" (UniqueName: \"kubernetes.io/projected/5899d096-c8b6-4b3c-979c-45e981e5acc8-kube-api-access-tc8sz\") pod \"maas-api-revoke-sub-79805ca9-9zkb7\" (UID: \"5899d096-c8b6-4b3c-979c-45e981e5acc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"
Oct 31 14:08:12.844362 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.844138    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5899d096-c8b6-4b3c-979c-45e981e5acc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9zkb7\" (UID: \"5899d096-c8b6-4b3c-979c-45e981e5acc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"
Oct 31 14:08:12.844362 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.844190    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5899d096-c8b6-4b3c-979c-45e981e5acc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9zkb7\" (UID: \"5899d096-c8b6-4b3c-979c-45e981e5acc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"
Oct 31 14:08:12.844857 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.844832    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5899d096-c8b6-4b3c-979c-45e981e5acc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9zkb7\" (UID: \"5899d096-c8b6-4b3c-979c-45e981e5acc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"
Oct 31 14:08:12.846765 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.846743    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5899d096-c8b6-4b3c-979c-45e981e5acc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9zkb7\" (UID: \"5899d096-c8b6-4b3c-979c-45e981e5acc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"
Oct 31 14:08:12.858823 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.858797    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc8sz\" (UniqueName: \"kubernetes.io/projected/5899d096-c8b6-4b3c-979c-45e981e5acc8-kube-api-access-tc8sz\") pod \"maas-api-revoke-sub-79805ca9-9zkb7\" (UID: \"5899d096-c8b6-4b3c-979c-45e981e5acc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"
Oct 31 14:08:12.961834 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:12.961802    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"
Oct 31 14:08:13.249434 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:13.249395    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"]
Oct 31 14:08:13.251560 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:08:13.251531    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5899d096_c8b6_4b3c_979c_45e981e5acc8.slice/crio-863f456b72f22fe85756d4357b28e68849d43deb974dc4d81fb966fe70ab91a9 WatchSource:0}: Error finding container 863f456b72f22fe85756d4357b28e68849d43deb974dc4d81fb966fe70ab91a9: Status 404 returned error can't find the container with id 863f456b72f22fe85756d4357b28e68849d43deb974dc4d81fb966fe70ab91a9
Oct 31 14:08:14.100057 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:14.100012    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7" event={"ID":"5899d096-c8b6-4b3c-979c-45e981e5acc8","Type":"ContainerStarted","Data":"aff65769a450843c61a6f38f72cca608e2efc0ba9c6a2eba256032887439a4f9"}
Oct 31 14:08:14.100509 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:14.100094    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7" event={"ID":"5899d096-c8b6-4b3c-979c-45e981e5acc8","Type":"ContainerStarted","Data":"863f456b72f22fe85756d4357b28e68849d43deb974dc4d81fb966fe70ab91a9"}
Oct 31 14:08:14.122039 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:14.121987    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7" podStartSLOduration=2.121970838 podStartE2EDuration="2.121970838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:08:14.119690594 +0000 UTC m=+3891.518039219" watchObservedRunningTime="2026-10-31 14:08:14.121970838 +0000 UTC m=+3891.520319482"
Oct 31 14:08:15.106897 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:15.106844    4428 generic.go:356] "Generic (PLEG): container finished" podID="5899d096-c8b6-4b3c-979c-45e981e5acc8" containerID="aff65769a450843c61a6f38f72cca608e2efc0ba9c6a2eba256032887439a4f9" exitCode=0
Oct 31 14:08:15.107334 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:15.106928    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7" event={"ID":"5899d096-c8b6-4b3c-979c-45e981e5acc8","Type":"ContainerDied","Data":"aff65769a450843c61a6f38f72cca608e2efc0ba9c6a2eba256032887439a4f9"}
Oct 31 14:08:16.221317 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:16.221274    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:08:16.249799 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:16.249774    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"
Oct 31 14:08:16.275787 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:16.275758    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5899d096-c8b6-4b3c-979c-45e981e5acc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5899d096-c8b6-4b3c-979c-45e981e5acc8-maas-api-xks-ca\") pod \"5899d096-c8b6-4b3c-979c-45e981e5acc8\" (UID: \"5899d096-c8b6-4b3c-979c-45e981e5acc8\") "
Oct 31 14:08:16.275972 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:16.275954    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5899d096-c8b6-4b3c-979c-45e981e5acc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5899d096-c8b6-4b3c-979c-45e981e5acc8-maas-api-openshift-ca\") pod \"5899d096-c8b6-4b3c-979c-45e981e5acc8\" (UID: \"5899d096-c8b6-4b3c-979c-45e981e5acc8\") "
Oct 31 14:08:16.276029 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:16.275996    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5899d096-c8b6-4b3c-979c-45e981e5acc8-kube-api-access-tc8sz\" (UniqueName: \"kubernetes.io/projected/5899d096-c8b6-4b3c-979c-45e981e5acc8-kube-api-access-tc8sz\") pod \"5899d096-c8b6-4b3c-979c-45e981e5acc8\" (UID: \"5899d096-c8b6-4b3c-979c-45e981e5acc8\") "
Oct 31 14:08:16.276364 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:16.276335    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5899d096-c8b6-4b3c-979c-45e981e5acc8-maas-api-openshift-ca" pod "5899d096-c8b6-4b3c-979c-45e981e5acc8" (UID: "5899d096-c8b6-4b3c-979c-45e981e5acc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:08:16.278251 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:16.278211    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5899d096-c8b6-4b3c-979c-45e981e5acc8-maas-api-xks-ca" pod "5899d096-c8b6-4b3c-979c-45e981e5acc8" (UID: "5899d096-c8b6-4b3c-979c-45e981e5acc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:08:16.278369 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:16.278288    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5899d096-c8b6-4b3c-979c-45e981e5acc8-kube-api-access-tc8sz" pod "5899d096-c8b6-4b3c-979c-45e981e5acc8" (UID: "5899d096-c8b6-4b3c-979c-45e981e5acc8"). InnerVolumeSpecName "kube-api-access-tc8sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:08:16.377918 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:16.377782    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5899d096-c8b6-4b3c-979c-45e981e5acc8-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:08:16.377918 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:16.377819    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5899d096-c8b6-4b3c-979c-45e981e5acc8-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:08:16.377918 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:16.377829    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc8sz\" (UniqueName: \"kubernetes.io/projected/5899d096-c8b6-4b3c-979c-45e981e5acc8-kube-api-access-tc8sz\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:08:16.815421 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:16.815387    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:08:17.119320 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:17.119225    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"
Oct 31 14:08:17.119320 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:17.119269    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7" event={"ID":"5899d096-c8b6-4b3c-979c-45e981e5acc8","Type":"ContainerDied","Data":"863f456b72f22fe85756d4357b28e68849d43deb974dc4d81fb966fe70ab91a9"}
Oct 31 14:08:17.119545 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:17.119331    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="863f456b72f22fe85756d4357b28e68849d43deb974dc4d81fb966fe70ab91a9"
Oct 31 14:08:18.740630 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:08:18.740592    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:08:30.498138 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.498098    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"]
Oct 31 14:08:30.498736 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.498719    4428 cpu_manager.go:414] "containerMap: removing container" podUID="5899d096-c8b6-4b3c-979c-45e981e5acc8" containerName="revoke-keys"
Oct 31 14:08:30.498787 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.498738    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5899d096-c8b6-4b3c-979c-45e981e5acc8" containerName="revoke-keys"
Oct 31 14:08:30.498831 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.498821    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5899d096-c8b6-4b3c-979c-45e981e5acc8" containerName="revoke-keys"
Oct 31 14:08:30.502455 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.502435    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"
Oct 31 14:08:30.507548 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.507523    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:08:30.507690 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.507677    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:08:30.510603 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.510515    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"]
Oct 31 14:08:30.643128 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.643093    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91575831-49a4-41c4-842b-34c7f6fceb3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7mn22\" (UID: \"91575831-49a4-41c4-842b-34c7f6fceb3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"
Oct 31 14:08:30.643367 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.643182    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91575831-49a4-41c4-842b-34c7f6fceb3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7mn22\" (UID: \"91575831-49a4-41c4-842b-34c7f6fceb3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"
Oct 31 14:08:30.643367 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.643246    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2bwd\" (UniqueName: \"kubernetes.io/projected/91575831-49a4-41c4-842b-34c7f6fceb3c-kube-api-access-q2bwd\") pod \"maas-api-revoke-sub-0b7e8edb-7mn22\" (UID: \"91575831-49a4-41c4-842b-34c7f6fceb3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"
Oct 31 14:08:30.743761 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.743722    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91575831-49a4-41c4-842b-34c7f6fceb3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7mn22\" (UID: \"91575831-49a4-41c4-842b-34c7f6fceb3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"
Oct 31 14:08:30.743968 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.743769    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2bwd\" (UniqueName: \"kubernetes.io/projected/91575831-49a4-41c4-842b-34c7f6fceb3c-kube-api-access-q2bwd\") pod \"maas-api-revoke-sub-0b7e8edb-7mn22\" (UID: \"91575831-49a4-41c4-842b-34c7f6fceb3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"
Oct 31 14:08:30.743968 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.743820    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91575831-49a4-41c4-842b-34c7f6fceb3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7mn22\" (UID: \"91575831-49a4-41c4-842b-34c7f6fceb3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"
Oct 31 14:08:30.744559 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.744537    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91575831-49a4-41c4-842b-34c7f6fceb3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7mn22\" (UID: \"91575831-49a4-41c4-842b-34c7f6fceb3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"
Oct 31 14:08:30.746349 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.746330    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91575831-49a4-41c4-842b-34c7f6fceb3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7mn22\" (UID: \"91575831-49a4-41c4-842b-34c7f6fceb3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"
Oct 31 14:08:30.757558 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.757491    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2bwd\" (UniqueName: \"kubernetes.io/projected/91575831-49a4-41c4-842b-34c7f6fceb3c-kube-api-access-q2bwd\") pod \"maas-api-revoke-sub-0b7e8edb-7mn22\" (UID: \"91575831-49a4-41c4-842b-34c7f6fceb3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"
Oct 31 14:08:30.821382 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:30.821348    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"
Oct 31 14:08:31.108313 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:31.108255    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"]
Oct 31 14:08:31.110815 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:08:31.110787    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91575831_49a4_41c4_842b_34c7f6fceb3c.slice/crio-9f7f890ff2b7b1fb0e53704b66b5bc60dd0eb4a6fd17a76e3942acb5b46a61b3 WatchSource:0}: Error finding container 9f7f890ff2b7b1fb0e53704b66b5bc60dd0eb4a6fd17a76e3942acb5b46a61b3: Status 404 returned error can't find the container with id 9f7f890ff2b7b1fb0e53704b66b5bc60dd0eb4a6fd17a76e3942acb5b46a61b3
Oct 31 14:08:31.220943 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:31.220897    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22" event={"ID":"91575831-49a4-41c4-842b-34c7f6fceb3c","Type":"ContainerStarted","Data":"9f7f890ff2b7b1fb0e53704b66b5bc60dd0eb4a6fd17a76e3942acb5b46a61b3"}
Oct 31 14:08:32.227988 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:32.227950    4428 generic.go:356] "Generic (PLEG): container finished" podID="91575831-49a4-41c4-842b-34c7f6fceb3c" containerID="0ad9c867a4786524becd59c36ee228022ea158c7657e08a15be8b94f747cab8d" exitCode=0
Oct 31 14:08:32.228453 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:32.228037    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22" event={"ID":"91575831-49a4-41c4-842b-34c7f6fceb3c","Type":"ContainerDied","Data":"0ad9c867a4786524becd59c36ee228022ea158c7657e08a15be8b94f747cab8d"}
Oct 31 14:08:33.369306 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:33.369276    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"
Oct 31 14:08:33.466938 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:33.466904    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91575831-49a4-41c4-842b-34c7f6fceb3c-kube-api-access-q2bwd\" (UniqueName: \"kubernetes.io/projected/91575831-49a4-41c4-842b-34c7f6fceb3c-kube-api-access-q2bwd\") pod \"91575831-49a4-41c4-842b-34c7f6fceb3c\" (UID: \"91575831-49a4-41c4-842b-34c7f6fceb3c\") "
Oct 31 14:08:33.467154 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:33.467041    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91575831-49a4-41c4-842b-34c7f6fceb3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91575831-49a4-41c4-842b-34c7f6fceb3c-maas-api-xks-ca\") pod \"91575831-49a4-41c4-842b-34c7f6fceb3c\" (UID: \"91575831-49a4-41c4-842b-34c7f6fceb3c\") "
Oct 31 14:08:33.467154 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:33.467069    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91575831-49a4-41c4-842b-34c7f6fceb3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91575831-49a4-41c4-842b-34c7f6fceb3c-maas-api-openshift-ca\") pod \"91575831-49a4-41c4-842b-34c7f6fceb3c\" (UID: \"91575831-49a4-41c4-842b-34c7f6fceb3c\") "
Oct 31 14:08:33.467531 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:33.467490    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91575831-49a4-41c4-842b-34c7f6fceb3c-maas-api-openshift-ca" pod "91575831-49a4-41c4-842b-34c7f6fceb3c" (UID: "91575831-49a4-41c4-842b-34c7f6fceb3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:08:33.469505 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:33.469452    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91575831-49a4-41c4-842b-34c7f6fceb3c-kube-api-access-q2bwd" pod "91575831-49a4-41c4-842b-34c7f6fceb3c" (UID: "91575831-49a4-41c4-842b-34c7f6fceb3c"). InnerVolumeSpecName "kube-api-access-q2bwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:08:33.469505 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:33.469452    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91575831-49a4-41c4-842b-34c7f6fceb3c-maas-api-xks-ca" pod "91575831-49a4-41c4-842b-34c7f6fceb3c" (UID: "91575831-49a4-41c4-842b-34c7f6fceb3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:08:33.568117 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:33.568009    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91575831-49a4-41c4-842b-34c7f6fceb3c-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:08:33.568117 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:33.568047    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91575831-49a4-41c4-842b-34c7f6fceb3c-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:08:33.568117 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:33.568063    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2bwd\" (UniqueName: \"kubernetes.io/projected/91575831-49a4-41c4-842b-34c7f6fceb3c-kube-api-access-q2bwd\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:08:34.240208 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:34.240129    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"
Oct 31 14:08:34.240393 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:34.240123    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22" event={"ID":"91575831-49a4-41c4-842b-34c7f6fceb3c","Type":"ContainerDied","Data":"9f7f890ff2b7b1fb0e53704b66b5bc60dd0eb4a6fd17a76e3942acb5b46a61b3"}
Oct 31 14:08:34.240393 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:34.240269    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f7f890ff2b7b1fb0e53704b66b5bc60dd0eb4a6fd17a76e3942acb5b46a61b3"
Oct 31 14:08:36.014214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:36.014181    4428 scope.go:122] "RemoveContainer" containerID="eb4721a55ea2f0ddbeae07855741ace0ddca3a2b11d998478e8bb16c873e3d44"
Oct 31 14:08:36.023802 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:36.023779    4428 scope.go:122] "RemoveContainer" containerID="8ea423ff03676f42fa3c5aed52140c2e7ef08b9ab5196f8ee180cdfb9a3ecc66"
Oct 31 14:08:36.032962 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:36.032934    4428 scope.go:122] "RemoveContainer" containerID="3011b44c1a90a3f60c8b17d31b4ad11c6a379c715b5e02d8e4f7ea322f6d0fa8"
Oct 31 14:08:36.041547 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:36.041528    4428 scope.go:122] "RemoveContainer" containerID="5584440e5b9fe1f5d1d91cb1b99b855a00e4eaf4fe4d35d893d85f4d7abc4670"
Oct 31 14:08:36.050002 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:36.049981    4428 scope.go:122] "RemoveContainer" containerID="8d436e33b5ef9763adffba5d0c4e0cde158ec039ae1fe256943316603601168b"
Oct 31 14:08:36.058627 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:36.058610    4428 scope.go:122] "RemoveContainer" containerID="f05921fc7bd367af33f15a33ed942510d44f4d68026ee3fadeeb2b28787c3a6f"
Oct 31 14:08:36.066936 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:36.066912    4428 scope.go:122] "RemoveContainer" containerID="14626cc75a43f4e592fde67eabaefd0e3ec029dda77b8645dd16cbdbf08fefbb"
Oct 31 14:08:36.075392 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:36.075369    4428 scope.go:122] "RemoveContainer" containerID="ac46682da3634950d34eab0960508be33f5a3f577597626f2007ed1fd66bf4ea"
Oct 31 14:08:41.188023 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.187984    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"]
Oct 31 14:08:41.188621 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.188599    4428 cpu_manager.go:414] "containerMap: removing container" podUID="91575831-49a4-41c4-842b-34c7f6fceb3c" containerName="revoke-keys"
Oct 31 14:08:41.188705 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.188625    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91575831-49a4-41c4-842b-34c7f6fceb3c" containerName="revoke-keys"
Oct 31 14:08:41.188775 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.188734    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="91575831-49a4-41c4-842b-34c7f6fceb3c" containerName="revoke-keys"
Oct 31 14:08:41.194404 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.194380    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"
Oct 31 14:08:41.199793 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.199765    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:08:41.200083 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.200063    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:08:41.202527 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.202497    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"]
Oct 31 14:08:41.340115 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.340074    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzp2b\" (UniqueName: \"kubernetes.io/projected/4e3ab550-7f02-41d4-a74a-2fa9683610fd-kube-api-access-hzp2b\") pod \"maas-api-revoke-sub-3d4156f1-7rmk9\" (UID: \"4e3ab550-7f02-41d4-a74a-2fa9683610fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"
Oct 31 14:08:41.340311 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.340146    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e3ab550-7f02-41d4-a74a-2fa9683610fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7rmk9\" (UID: \"4e3ab550-7f02-41d4-a74a-2fa9683610fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"
Oct 31 14:08:41.340311 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.340251    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e3ab550-7f02-41d4-a74a-2fa9683610fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7rmk9\" (UID: \"4e3ab550-7f02-41d4-a74a-2fa9683610fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"
Oct 31 14:08:41.440944 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.440815    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzp2b\" (UniqueName: \"kubernetes.io/projected/4e3ab550-7f02-41d4-a74a-2fa9683610fd-kube-api-access-hzp2b\") pod \"maas-api-revoke-sub-3d4156f1-7rmk9\" (UID: \"4e3ab550-7f02-41d4-a74a-2fa9683610fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"
Oct 31 14:08:41.441159 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.440945    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e3ab550-7f02-41d4-a74a-2fa9683610fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7rmk9\" (UID: \"4e3ab550-7f02-41d4-a74a-2fa9683610fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"
Oct 31 14:08:41.441159 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.440982    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e3ab550-7f02-41d4-a74a-2fa9683610fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7rmk9\" (UID: \"4e3ab550-7f02-41d4-a74a-2fa9683610fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"
Oct 31 14:08:41.441621 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.441596    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e3ab550-7f02-41d4-a74a-2fa9683610fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7rmk9\" (UID: \"4e3ab550-7f02-41d4-a74a-2fa9683610fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"
Oct 31 14:08:41.443656 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.443630    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e3ab550-7f02-41d4-a74a-2fa9683610fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7rmk9\" (UID: \"4e3ab550-7f02-41d4-a74a-2fa9683610fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"
Oct 31 14:08:41.455151 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.455117    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzp2b\" (UniqueName: \"kubernetes.io/projected/4e3ab550-7f02-41d4-a74a-2fa9683610fd-kube-api-access-hzp2b\") pod \"maas-api-revoke-sub-3d4156f1-7rmk9\" (UID: \"4e3ab550-7f02-41d4-a74a-2fa9683610fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"
Oct 31 14:08:41.510434 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.510389    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"
Oct 31 14:08:41.796130 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:41.796102    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"]
Oct 31 14:08:41.798318 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:08:41.798290    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3ab550_7f02_41d4_a74a_2fa9683610fd.slice/crio-8fe6a24c575402b81dd92475f39b3e1b7dc82c963595e0a25e9e93ce0b21eaff WatchSource:0}: Error finding container 8fe6a24c575402b81dd92475f39b3e1b7dc82c963595e0a25e9e93ce0b21eaff: Status 404 returned error can't find the container with id 8fe6a24c575402b81dd92475f39b3e1b7dc82c963595e0a25e9e93ce0b21eaff
Oct 31 14:08:42.282750 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:42.282707    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9" event={"ID":"4e3ab550-7f02-41d4-a74a-2fa9683610fd","Type":"ContainerStarted","Data":"4c83dff49f65f9a67a6ce50444f5cfbec6a130f2b006457df5051da37f5d2cd1"}
Oct 31 14:08:42.283191 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:42.282764    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9" event={"ID":"4e3ab550-7f02-41d4-a74a-2fa9683610fd","Type":"ContainerStarted","Data":"8fe6a24c575402b81dd92475f39b3e1b7dc82c963595e0a25e9e93ce0b21eaff"}
Oct 31 14:08:42.303391 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:42.303300    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9" podStartSLOduration=1.303285298 podStartE2EDuration="1.303285298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:08:42.301280387 +0000 UTC m=+3919.699629011" watchObservedRunningTime="2026-10-31 14:08:42.303285298 +0000 UTC m=+3919.701633922"
Oct 31 14:08:43.289882 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:43.289827    4428 generic.go:356] "Generic (PLEG): container finished" podID="4e3ab550-7f02-41d4-a74a-2fa9683610fd" containerID="4c83dff49f65f9a67a6ce50444f5cfbec6a130f2b006457df5051da37f5d2cd1" exitCode=0
Oct 31 14:08:43.290319 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:43.289907    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9" event={"ID":"4e3ab550-7f02-41d4-a74a-2fa9683610fd","Type":"ContainerDied","Data":"4c83dff49f65f9a67a6ce50444f5cfbec6a130f2b006457df5051da37f5d2cd1"}
Oct 31 14:08:44.436051 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:44.436023    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"
Oct 31 14:08:44.572593 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:44.572500    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3ab550-7f02-41d4-a74a-2fa9683610fd-kube-api-access-hzp2b\" (UniqueName: \"kubernetes.io/projected/4e3ab550-7f02-41d4-a74a-2fa9683610fd-kube-api-access-hzp2b\") pod \"4e3ab550-7f02-41d4-a74a-2fa9683610fd\" (UID: \"4e3ab550-7f02-41d4-a74a-2fa9683610fd\") "
Oct 31 14:08:44.572593 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:44.572580    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e3ab550-7f02-41d4-a74a-2fa9683610fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e3ab550-7f02-41d4-a74a-2fa9683610fd-maas-api-openshift-ca\") pod \"4e3ab550-7f02-41d4-a74a-2fa9683610fd\" (UID: \"4e3ab550-7f02-41d4-a74a-2fa9683610fd\") "
Oct 31 14:08:44.572819 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:44.572612    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e3ab550-7f02-41d4-a74a-2fa9683610fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e3ab550-7f02-41d4-a74a-2fa9683610fd-maas-api-xks-ca\") pod \"4e3ab550-7f02-41d4-a74a-2fa9683610fd\" (UID: \"4e3ab550-7f02-41d4-a74a-2fa9683610fd\") "
Oct 31 14:08:44.572971 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:44.572950    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3ab550-7f02-41d4-a74a-2fa9683610fd-maas-api-openshift-ca" pod "4e3ab550-7f02-41d4-a74a-2fa9683610fd" (UID: "4e3ab550-7f02-41d4-a74a-2fa9683610fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:08:44.574968 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:44.574944    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e3ab550-7f02-41d4-a74a-2fa9683610fd-maas-api-xks-ca" pod "4e3ab550-7f02-41d4-a74a-2fa9683610fd" (UID: "4e3ab550-7f02-41d4-a74a-2fa9683610fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:08:44.575054 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:44.574944    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3ab550-7f02-41d4-a74a-2fa9683610fd-kube-api-access-hzp2b" pod "4e3ab550-7f02-41d4-a74a-2fa9683610fd" (UID: "4e3ab550-7f02-41d4-a74a-2fa9683610fd"). InnerVolumeSpecName "kube-api-access-hzp2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:08:44.674229 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:44.674194    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzp2b\" (UniqueName: \"kubernetes.io/projected/4e3ab550-7f02-41d4-a74a-2fa9683610fd-kube-api-access-hzp2b\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:08:44.674229 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:44.674225    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e3ab550-7f02-41d4-a74a-2fa9683610fd-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:08:44.674229 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:44.674235    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e3ab550-7f02-41d4-a74a-2fa9683610fd-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:08:45.301672 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:45.301632    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9" event={"ID":"4e3ab550-7f02-41d4-a74a-2fa9683610fd","Type":"ContainerDied","Data":"8fe6a24c575402b81dd92475f39b3e1b7dc82c963595e0a25e9e93ce0b21eaff"}
Oct 31 14:08:45.301672 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:45.301652    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"
Oct 31 14:08:45.301942 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:45.301684    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fe6a24c575402b81dd92475f39b3e1b7dc82c963595e0a25e9e93ce0b21eaff"
Oct 31 14:08:48.519350 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:08:48.519305    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:08:48.740235 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:08:48.740192    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:09:02.489727 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.489674    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"]
Oct 31 14:09:02.492672 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.492643    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7rmk9"]
Oct 31 14:09:02.517099 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.517053    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"]
Oct 31 14:09:02.517848 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.517820    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4e3ab550-7f02-41d4-a74a-2fa9683610fd" containerName="revoke-keys"
Oct 31 14:09:02.517933 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.517853    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3ab550-7f02-41d4-a74a-2fa9683610fd" containerName="revoke-keys"
Oct 31 14:09:02.518023 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.518007    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3ab550-7f02-41d4-a74a-2fa9683610fd" containerName="revoke-keys"
Oct 31 14:09:02.522109 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.522079    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"
Oct 31 14:09:02.526895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.526842    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:09:02.527074 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.526975    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:09:02.529327 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.529299    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"]
Oct 31 14:09:02.571487 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.571455    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99c47b7-3b46-4bd3-9628-100441acaa3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cqs62\" (UID: \"e99c47b7-3b46-4bd3-9628-100441acaa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"
Oct 31 14:09:02.571705 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.571500    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99c47b7-3b46-4bd3-9628-100441acaa3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cqs62\" (UID: \"e99c47b7-3b46-4bd3-9628-100441acaa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"
Oct 31 14:09:02.571705 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.571533    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqfwx\" (UniqueName: \"kubernetes.io/projected/e99c47b7-3b46-4bd3-9628-100441acaa3f-kube-api-access-lqfwx\") pod \"maas-api-revoke-sub-3d4156f1-cqs62\" (UID: \"e99c47b7-3b46-4bd3-9628-100441acaa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"
Oct 31 14:09:02.672318 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.672274    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99c47b7-3b46-4bd3-9628-100441acaa3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cqs62\" (UID: \"e99c47b7-3b46-4bd3-9628-100441acaa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"
Oct 31 14:09:02.672551 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.672329    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99c47b7-3b46-4bd3-9628-100441acaa3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cqs62\" (UID: \"e99c47b7-3b46-4bd3-9628-100441acaa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"
Oct 31 14:09:02.672551 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.672363    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqfwx\" (UniqueName: \"kubernetes.io/projected/e99c47b7-3b46-4bd3-9628-100441acaa3f-kube-api-access-lqfwx\") pod \"maas-api-revoke-sub-3d4156f1-cqs62\" (UID: \"e99c47b7-3b46-4bd3-9628-100441acaa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"
Oct 31 14:09:02.673077 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.673054    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99c47b7-3b46-4bd3-9628-100441acaa3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cqs62\" (UID: \"e99c47b7-3b46-4bd3-9628-100441acaa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"
Oct 31 14:09:02.674861 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.674841    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99c47b7-3b46-4bd3-9628-100441acaa3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cqs62\" (UID: \"e99c47b7-3b46-4bd3-9628-100441acaa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"
Oct 31 14:09:02.686379 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.686350    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqfwx\" (UniqueName: \"kubernetes.io/projected/e99c47b7-3b46-4bd3-9628-100441acaa3f-kube-api-access-lqfwx\") pod \"maas-api-revoke-sub-3d4156f1-cqs62\" (UID: \"e99c47b7-3b46-4bd3-9628-100441acaa3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"
Oct 31 14:09:02.753543 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.753459    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e3ab550-7f02-41d4-a74a-2fa9683610fd" path="/var/lib/kubelet/pods/4e3ab550-7f02-41d4-a74a-2fa9683610fd/volumes"
Oct 31 14:09:02.842958 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:02.842922    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"
Oct 31 14:09:03.124056 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:03.123976    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"]
Oct 31 14:09:03.127857 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:09:03.127826    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99c47b7_3b46_4bd3_9628_100441acaa3f.slice/crio-3d1ca5214bace54827100c47efbfba513d01b3642b719fac3a4a7a95a3e3a762 WatchSource:0}: Error finding container 3d1ca5214bace54827100c47efbfba513d01b3642b719fac3a4a7a95a3e3a762: Status 404 returned error can't find the container with id 3d1ca5214bace54827100c47efbfba513d01b3642b719fac3a4a7a95a3e3a762
Oct 31 14:09:03.440759 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:03.440662    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62" event={"ID":"e99c47b7-3b46-4bd3-9628-100441acaa3f","Type":"ContainerStarted","Data":"3d1ca5214bace54827100c47efbfba513d01b3642b719fac3a4a7a95a3e3a762"}
Oct 31 14:09:04.448795 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:04.448757    4428 generic.go:356] "Generic (PLEG): container finished" podID="e99c47b7-3b46-4bd3-9628-100441acaa3f" containerID="8053a305f7efeea7e5d061b53a59e7ce1e264dbb4003fb310b391b2955b6a415" exitCode=0
Oct 31 14:09:04.449222 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:04.448892    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62" event={"ID":"e99c47b7-3b46-4bd3-9628-100441acaa3f","Type":"ContainerDied","Data":"8053a305f7efeea7e5d061b53a59e7ce1e264dbb4003fb310b391b2955b6a415"}
Oct 31 14:09:05.599160 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:05.599136    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"
Oct 31 14:09:05.701860 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:05.701822    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e99c47b7-3b46-4bd3-9628-100441acaa3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99c47b7-3b46-4bd3-9628-100441acaa3f-maas-api-xks-ca\") pod \"e99c47b7-3b46-4bd3-9628-100441acaa3f\" (UID: \"e99c47b7-3b46-4bd3-9628-100441acaa3f\") "
Oct 31 14:09:05.702065 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:05.701904    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99c47b7-3b46-4bd3-9628-100441acaa3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99c47b7-3b46-4bd3-9628-100441acaa3f-maas-api-openshift-ca\") pod \"e99c47b7-3b46-4bd3-9628-100441acaa3f\" (UID: \"e99c47b7-3b46-4bd3-9628-100441acaa3f\") "
Oct 31 14:09:05.702065 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:05.701951    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99c47b7-3b46-4bd3-9628-100441acaa3f-kube-api-access-lqfwx\" (UniqueName: \"kubernetes.io/projected/e99c47b7-3b46-4bd3-9628-100441acaa3f-kube-api-access-lqfwx\") pod \"e99c47b7-3b46-4bd3-9628-100441acaa3f\" (UID: \"e99c47b7-3b46-4bd3-9628-100441acaa3f\") "
Oct 31 14:09:05.702279 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:05.702253    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99c47b7-3b46-4bd3-9628-100441acaa3f-maas-api-openshift-ca" pod "e99c47b7-3b46-4bd3-9628-100441acaa3f" (UID: "e99c47b7-3b46-4bd3-9628-100441acaa3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:09:05.704226 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:05.704196    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e99c47b7-3b46-4bd3-9628-100441acaa3f-maas-api-xks-ca" pod "e99c47b7-3b46-4bd3-9628-100441acaa3f" (UID: "e99c47b7-3b46-4bd3-9628-100441acaa3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:09:05.704367 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:05.704275    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99c47b7-3b46-4bd3-9628-100441acaa3f-kube-api-access-lqfwx" pod "e99c47b7-3b46-4bd3-9628-100441acaa3f" (UID: "e99c47b7-3b46-4bd3-9628-100441acaa3f"). InnerVolumeSpecName "kube-api-access-lqfwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:09:05.803475 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:05.803427    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99c47b7-3b46-4bd3-9628-100441acaa3f-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:09:05.803475 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:05.803464    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99c47b7-3b46-4bd3-9628-100441acaa3f-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:09:05.803475 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:05.803480    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqfwx\" (UniqueName: \"kubernetes.io/projected/e99c47b7-3b46-4bd3-9628-100441acaa3f-kube-api-access-lqfwx\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:09:06.460883 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:06.460839    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62" event={"ID":"e99c47b7-3b46-4bd3-9628-100441acaa3f","Type":"ContainerDied","Data":"3d1ca5214bace54827100c47efbfba513d01b3642b719fac3a4a7a95a3e3a762"}
Oct 31 14:09:06.461117 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:06.460922    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d1ca5214bace54827100c47efbfba513d01b3642b719fac3a4a7a95a3e3a762"
Oct 31 14:09:06.461117 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:06.460851    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"
Oct 31 14:09:16.131243 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:16.131207    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:09:16.286623 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:16.286581    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:09:18.741537 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:09:18.741492    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:09:28.075331 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.075291    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"]
Oct 31 14:09:28.075749 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.075715    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e99c47b7-3b46-4bd3-9628-100441acaa3f" containerName="revoke-keys"
Oct 31 14:09:28.075749 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.075728    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99c47b7-3b46-4bd3-9628-100441acaa3f" containerName="revoke-keys"
Oct 31 14:09:28.075830 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.075802    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99c47b7-3b46-4bd3-9628-100441acaa3f" containerName="revoke-keys"
Oct 31 14:09:28.079258 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.079240    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"
Oct 31 14:09:28.084188 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.084162    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:09:28.084341 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.084207    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:09:28.092513 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.092151    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"]
Oct 31 14:09:28.248964 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.248917    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ngjxs\" (UID: \"ba825ec5-dc91-4ffd-81cb-b91da48e83ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"
Oct 31 14:09:28.249185 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.248994    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4rsc\" (UniqueName: \"kubernetes.io/projected/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-kube-api-access-m4rsc\") pod \"maas-api-revoke-sub-e2fc9c5b-ngjxs\" (UID: \"ba825ec5-dc91-4ffd-81cb-b91da48e83ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"
Oct 31 14:09:28.249185 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.249089    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ngjxs\" (UID: \"ba825ec5-dc91-4ffd-81cb-b91da48e83ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"
Oct 31 14:09:28.349895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.349762    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ngjxs\" (UID: \"ba825ec5-dc91-4ffd-81cb-b91da48e83ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"
Oct 31 14:09:28.349895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.349846    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4rsc\" (UniqueName: \"kubernetes.io/projected/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-kube-api-access-m4rsc\") pod \"maas-api-revoke-sub-e2fc9c5b-ngjxs\" (UID: \"ba825ec5-dc91-4ffd-81cb-b91da48e83ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"
Oct 31 14:09:28.350117 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.349960    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ngjxs\" (UID: \"ba825ec5-dc91-4ffd-81cb-b91da48e83ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"
Oct 31 14:09:28.350495 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.350473    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ngjxs\" (UID: \"ba825ec5-dc91-4ffd-81cb-b91da48e83ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"
Oct 31 14:09:28.352514 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.352492    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ngjxs\" (UID: \"ba825ec5-dc91-4ffd-81cb-b91da48e83ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"
Oct 31 14:09:28.363657 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.363614    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4rsc\" (UniqueName: \"kubernetes.io/projected/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-kube-api-access-m4rsc\") pod \"maas-api-revoke-sub-e2fc9c5b-ngjxs\" (UID: \"ba825ec5-dc91-4ffd-81cb-b91da48e83ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"
Oct 31 14:09:28.399058 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.399020    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"
Oct 31 14:09:28.685859 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:28.685718    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"]
Oct 31 14:09:28.688368 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:09:28.688333    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba825ec5_dc91_4ffd_81cb_b91da48e83ba.slice/crio-47a0c58a99b40cb986c4dd8358972d433ab458c51a167b102391d3ec150dec6b WatchSource:0}: Error finding container 47a0c58a99b40cb986c4dd8358972d433ab458c51a167b102391d3ec150dec6b: Status 404 returned error can't find the container with id 47a0c58a99b40cb986c4dd8358972d433ab458c51a167b102391d3ec150dec6b
Oct 31 14:09:29.615776 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:29.615736    4428 generic.go:356] "Generic (PLEG): container finished" podID="ba825ec5-dc91-4ffd-81cb-b91da48e83ba" containerID="654e3a0d1b61fce3bd8723eea985a862e21afc7bcdc6dee2559082d433385b8d" exitCode=0
Oct 31 14:09:29.616249 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:29.615801    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs" event={"ID":"ba825ec5-dc91-4ffd-81cb-b91da48e83ba","Type":"ContainerDied","Data":"654e3a0d1b61fce3bd8723eea985a862e21afc7bcdc6dee2559082d433385b8d"}
Oct 31 14:09:29.616249 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:29.615850    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs" event={"ID":"ba825ec5-dc91-4ffd-81cb-b91da48e83ba","Type":"ContainerStarted","Data":"47a0c58a99b40cb986c4dd8358972d433ab458c51a167b102391d3ec150dec6b"}
Oct 31 14:09:30.762672 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:30.762645    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"
Oct 31 14:09:30.874948 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:30.874814    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-maas-api-xks-ca\") pod \"ba825ec5-dc91-4ffd-81cb-b91da48e83ba\" (UID: \"ba825ec5-dc91-4ffd-81cb-b91da48e83ba\") "
Oct 31 14:09:30.874948 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:30.874908    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-kube-api-access-m4rsc\" (UniqueName: \"kubernetes.io/projected/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-kube-api-access-m4rsc\") pod \"ba825ec5-dc91-4ffd-81cb-b91da48e83ba\" (UID: \"ba825ec5-dc91-4ffd-81cb-b91da48e83ba\") "
Oct 31 14:09:30.874948 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:30.874945    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-maas-api-openshift-ca\") pod \"ba825ec5-dc91-4ffd-81cb-b91da48e83ba\" (UID: \"ba825ec5-dc91-4ffd-81cb-b91da48e83ba\") "
Oct 31 14:09:30.875362 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:30.875333    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-maas-api-openshift-ca" pod "ba825ec5-dc91-4ffd-81cb-b91da48e83ba" (UID: "ba825ec5-dc91-4ffd-81cb-b91da48e83ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:09:30.877252 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:30.877226    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-maas-api-xks-ca" pod "ba825ec5-dc91-4ffd-81cb-b91da48e83ba" (UID: "ba825ec5-dc91-4ffd-81cb-b91da48e83ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:09:30.877355 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:30.877325    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-kube-api-access-m4rsc" pod "ba825ec5-dc91-4ffd-81cb-b91da48e83ba" (UID: "ba825ec5-dc91-4ffd-81cb-b91da48e83ba"). InnerVolumeSpecName "kube-api-access-m4rsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:09:30.975882 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:30.975827    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:09:30.975882 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:30.975855    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4rsc\" (UniqueName: \"kubernetes.io/projected/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-kube-api-access-m4rsc\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:09:30.975882 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:30.975878    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba825ec5-dc91-4ffd-81cb-b91da48e83ba-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:09:31.628055 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:31.628013    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs" event={"ID":"ba825ec5-dc91-4ffd-81cb-b91da48e83ba","Type":"ContainerDied","Data":"47a0c58a99b40cb986c4dd8358972d433ab458c51a167b102391d3ec150dec6b"}
Oct 31 14:09:31.628291 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:31.628080    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47a0c58a99b40cb986c4dd8358972d433ab458c51a167b102391d3ec150dec6b"
Oct 31 14:09:31.628291 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:31.628032    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"
Oct 31 14:09:36.132196 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:36.132163    4428 scope.go:122] "RemoveContainer" containerID="a24cd72626669a4cf831e93aaa0eb477e2270bbd4d94b1ca209e520fa6db1e42"
Oct 31 14:09:36.141564 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:36.141539    4428 scope.go:122] "RemoveContainer" containerID="fe54ba873db8d09cb569a20ac6be0fa85a31102331fa8410dc3dcae229bcb9bd"
Oct 31 14:09:36.150056 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:36.150031    4428 scope.go:122] "RemoveContainer" containerID="6cd47b9fa9538d288cc1124d4e141380a2e2ca62f0bc521262354f83dabada5d"
Oct 31 14:09:36.158201 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:36.158177    4428 scope.go:122] "RemoveContainer" containerID="c73a0f2f98ffd03940fc725cffbad8e9564bbc0f533f332d5419f7c3baa15b72"
Oct 31 14:09:36.166201 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:36.166179    4428 scope.go:122] "RemoveContainer" containerID="041a1b26ddad0a92363f412581e5e242fe5b916c89d54f054e6da5e3480650e6"
Oct 31 14:09:36.174190 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:36.174172    4428 scope.go:122] "RemoveContainer" containerID="ca3bc9dc2b8361179e7f548f47a82a0e00bddca315fc28f65e7737537cec79ee"
Oct 31 14:09:38.307584 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.307539    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"]
Oct 31 14:09:38.308202 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.308180    4428 cpu_manager.go:414] "containerMap: removing container" podUID="ba825ec5-dc91-4ffd-81cb-b91da48e83ba" containerName="revoke-keys"
Oct 31 14:09:38.308297 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.308203    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba825ec5-dc91-4ffd-81cb-b91da48e83ba" containerName="revoke-keys"
Oct 31 14:09:38.308363 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.308312    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba825ec5-dc91-4ffd-81cb-b91da48e83ba" containerName="revoke-keys"
Oct 31 14:09:38.313198 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.313175    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"
Oct 31 14:09:38.319062 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.319012    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:09:38.319264 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.319242    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:09:38.323289 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.322336    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"]
Oct 31 14:09:38.338459 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.338421    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-dqbng\" (UID: \"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"
Oct 31 14:09:38.338704 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.338599    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-dqbng\" (UID: \"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"
Oct 31 14:09:38.338704 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.338649    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvphk\" (UniqueName: \"kubernetes.io/projected/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-kube-api-access-dvphk\") pod \"maas-api-revoke-sub-5a29efb3-dqbng\" (UID: \"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"
Oct 31 14:09:38.439898 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.439832    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-dqbng\" (UID: \"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"
Oct 31 14:09:38.439898 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.439892    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvphk\" (UniqueName: \"kubernetes.io/projected/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-kube-api-access-dvphk\") pod \"maas-api-revoke-sub-5a29efb3-dqbng\" (UID: \"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"
Oct 31 14:09:38.440208 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.439947    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-dqbng\" (UID: \"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"
Oct 31 14:09:38.440625 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.440603    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-dqbng\" (UID: \"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"
Oct 31 14:09:38.442435 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.442412    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-dqbng\" (UID: \"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"
Oct 31 14:09:38.453361 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.453334    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvphk\" (UniqueName: \"kubernetes.io/projected/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-kube-api-access-dvphk\") pod \"maas-api-revoke-sub-5a29efb3-dqbng\" (UID: \"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"
Oct 31 14:09:38.629658 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.629565    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"
Oct 31 14:09:38.912603 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:38.912569    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"]
Oct 31 14:09:38.915731 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:09:38.915681    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb3c7b5_20fc_45c3_9b82_c2e3162f9fe9.slice/crio-eabc3b7ca898bceb18c175aaf35fa487203c83370db13009eabfd10aecc244c9 WatchSource:0}: Error finding container eabc3b7ca898bceb18c175aaf35fa487203c83370db13009eabfd10aecc244c9: Status 404 returned error can't find the container with id eabc3b7ca898bceb18c175aaf35fa487203c83370db13009eabfd10aecc244c9
Oct 31 14:09:39.681108 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:39.680991    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng" event={"ID":"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9","Type":"ContainerStarted","Data":"39c1dade2bcb2df30e8839580c09d36eb51794144cf6d71075cdee8477178106"}
Oct 31 14:09:39.681108 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:39.681070    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng" event={"ID":"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9","Type":"ContainerStarted","Data":"eabc3b7ca898bceb18c175aaf35fa487203c83370db13009eabfd10aecc244c9"}
Oct 31 14:09:39.704745 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:39.704687    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng" podStartSLOduration=1.704671567 podStartE2EDuration="1.704671567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:09:39.704307863 +0000 UTC m=+3977.102656488" watchObservedRunningTime="2026-10-31 14:09:39.704671567 +0000 UTC m=+3977.103020190"
Oct 31 14:09:40.715010 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:40.714946    4428 generic.go:356] "Generic (PLEG): container finished" podID="5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9" containerID="39c1dade2bcb2df30e8839580c09d36eb51794144cf6d71075cdee8477178106" exitCode=0
Oct 31 14:09:40.715595 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:40.715096    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng" event={"ID":"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9","Type":"ContainerDied","Data":"39c1dade2bcb2df30e8839580c09d36eb51794144cf6d71075cdee8477178106"}
Oct 31 14:09:41.953660 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:41.953347    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"
Oct 31 14:09:42.033500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:42.033402    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-kube-api-access-dvphk\" (UniqueName: \"kubernetes.io/projected/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-kube-api-access-dvphk\") pod \"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9\" (UID: \"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9\") "
Oct 31 14:09:42.033500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:42.033462    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-maas-api-xks-ca\") pod \"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9\" (UID: \"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9\") "
Oct 31 14:09:42.033730 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:42.033530    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-maas-api-openshift-ca\") pod \"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9\" (UID: \"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9\") "
Oct 31 14:09:42.034096 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:42.034068    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-maas-api-openshift-ca" pod "5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9" (UID: "5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:09:42.036115 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:42.036060    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-maas-api-xks-ca" pod "5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9" (UID: "5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:09:42.036270 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:42.036124    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-kube-api-access-dvphk" pod "5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9" (UID: "5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9"). InnerVolumeSpecName "kube-api-access-dvphk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:09:42.134835 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:42.134791    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:09:42.134835 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:42.134826    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvphk\" (UniqueName: \"kubernetes.io/projected/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-kube-api-access-dvphk\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:09:42.134835 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:42.134838    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:09:42.730620 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:42.730578    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"
Oct 31 14:09:42.730620 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:42.730602    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng" event={"ID":"5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9","Type":"ContainerDied","Data":"eabc3b7ca898bceb18c175aaf35fa487203c83370db13009eabfd10aecc244c9"}
Oct 31 14:09:42.730951 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:42.730649    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eabc3b7ca898bceb18c175aaf35fa487203c83370db13009eabfd10aecc244c9"
Oct 31 14:09:48.742767 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:09:48.742726    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:09:49.171518 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:49.171423    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:09:51.338244 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:51.338204    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:09:54.105790 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.105755    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"]
Oct 31 14:09:54.110575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.110544    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqs62"]
Oct 31 14:09:54.132745 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.132695    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"]
Oct 31 14:09:54.133539 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.133513    4428 cpu_manager.go:414] "containerMap: removing container" podUID="5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9" containerName="revoke-keys"
Oct 31 14:09:54.133539 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.133534    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9" containerName="revoke-keys"
Oct 31 14:09:54.133794 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.133696    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9" containerName="revoke-keys"
Oct 31 14:09:54.142361 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.142333    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"
Oct 31 14:09:54.146191 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.146164    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"]
Oct 31 14:09:54.147033 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.147009    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:09:54.147567 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.147546    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:09:54.258078 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.258037    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hqvl4\" (UID: \"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"
Oct 31 14:09:54.258078 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.258089    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hqvl4\" (UID: \"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"
Oct 31 14:09:54.258374 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.258248    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnjvt\" (UniqueName: \"kubernetes.io/projected/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-kube-api-access-jnjvt\") pod \"maas-api-revoke-sub-3d4156f1-hqvl4\" (UID: \"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"
Oct 31 14:09:54.359001 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.358899    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnjvt\" (UniqueName: \"kubernetes.io/projected/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-kube-api-access-jnjvt\") pod \"maas-api-revoke-sub-3d4156f1-hqvl4\" (UID: \"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"
Oct 31 14:09:54.359001 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.358973    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hqvl4\" (UID: \"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"
Oct 31 14:09:54.359257 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.359013    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hqvl4\" (UID: \"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"
Oct 31 14:09:54.359736 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.359695    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hqvl4\" (UID: \"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"
Oct 31 14:09:54.361608 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.361582    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hqvl4\" (UID: \"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"
Oct 31 14:09:54.372753 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.372731    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnjvt\" (UniqueName: \"kubernetes.io/projected/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-kube-api-access-jnjvt\") pod \"maas-api-revoke-sub-3d4156f1-hqvl4\" (UID: \"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"
Oct 31 14:09:54.458650 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.458610    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"
Oct 31 14:09:54.738502 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.738473    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"]
Oct 31 14:09:54.740120 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:09:54.740080    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcaa2b41_be05_4d2e_9b6d_ec3814a2b2e4.slice/crio-ad849da951f0903c46c857d8d5beececafec92848e68a7c7cffc690cfc021384 WatchSource:0}: Error finding container ad849da951f0903c46c857d8d5beececafec92848e68a7c7cffc690cfc021384: Status 404 returned error can't find the container with id ad849da951f0903c46c857d8d5beececafec92848e68a7c7cffc690cfc021384
Oct 31 14:09:54.754768 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.754733    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99c47b7-3b46-4bd3-9628-100441acaa3f" path="/var/lib/kubelet/pods/e99c47b7-3b46-4bd3-9628-100441acaa3f/volumes"
Oct 31 14:09:54.793254 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:54.793220    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4" event={"ID":"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4","Type":"ContainerStarted","Data":"ad849da951f0903c46c857d8d5beececafec92848e68a7c7cffc690cfc021384"}
Oct 31 14:09:55.799494 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:55.799451    4428 generic.go:356] "Generic (PLEG): container finished" podID="fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4" containerID="f499b613fcebbf6c206ee42a13b9def73660f9f296d68d3338ada5939688f3ce" exitCode=0
Oct 31 14:09:55.799936 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:55.799490    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4" event={"ID":"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4","Type":"ContainerDied","Data":"f499b613fcebbf6c206ee42a13b9def73660f9f296d68d3338ada5939688f3ce"}
Oct 31 14:09:56.943424 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:56.943398    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"
Oct 31 14:09:57.087337 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:57.087240    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-maas-api-openshift-ca\") pod \"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4\" (UID: \"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4\") "
Oct 31 14:09:57.087513 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:57.087383    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-kube-api-access-jnjvt\" (UniqueName: \"kubernetes.io/projected/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-kube-api-access-jnjvt\") pod \"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4\" (UID: \"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4\") "
Oct 31 14:09:57.087513 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:57.087420    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-maas-api-xks-ca\") pod \"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4\" (UID: \"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4\") "
Oct 31 14:09:57.087704 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:57.087672    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-maas-api-openshift-ca" pod "fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4" (UID: "fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:09:57.089753 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:57.089717    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-kube-api-access-jnjvt" pod "fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4" (UID: "fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4"). InnerVolumeSpecName "kube-api-access-jnjvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:09:57.089753 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:57.089729    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-maas-api-xks-ca" pod "fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4" (UID: "fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:09:57.188145 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:57.188106    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:09:57.188145 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:57.188138    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnjvt\" (UniqueName: \"kubernetes.io/projected/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-kube-api-access-jnjvt\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:09:57.188145 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:57.188151    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:09:57.815509 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:57.815479    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"
Oct 31 14:09:57.815704 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:57.815509    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4" event={"ID":"fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4","Type":"ContainerDied","Data":"ad849da951f0903c46c857d8d5beececafec92848e68a7c7cffc690cfc021384"}
Oct 31 14:09:57.815704 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:09:57.815561    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad849da951f0903c46c857d8d5beececafec92848e68a7c7cffc690cfc021384"
Oct 31 14:10:15.553643 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:15.553554    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:15.906226 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:15.906141    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"]
Oct 31 14:10:15.906787 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:15.906772    4428 cpu_manager.go:414] "containerMap: removing container" podUID="fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4" containerName="revoke-keys"
Oct 31 14:10:15.906840 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:15.906791    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4" containerName="revoke-keys"
Oct 31 14:10:15.906912 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:15.906902    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4" containerName="revoke-keys"
Oct 31 14:10:15.910780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:15.910754    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:15.915789 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:15.915766    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-c4292c-openshift-default-dockercfg-2t9s2\""
Oct 31 14:10:15.924084 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:15.924052    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"]
Oct 31 14:10:15.998321 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:15.998283    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-envoy\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:15.998321 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:15.998323    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-workload-certs\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:15.998596 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:15.998410    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-workload-socket\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:15.998596 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:15.998458    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61628be-a3de-4cb7-aba5-d31e11396ad5-istiod-ca-cert\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:15.998732 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:15.998605    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c4xb\" (UniqueName: \"kubernetes.io/projected/b61628be-a3de-4cb7-aba5-d31e11396ad5-kube-api-access-5c4xb\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:15.998732 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:15.998667    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-data\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:15.998732 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:15.998712    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-token\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:15.998915 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:15.998806    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-podinfo\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:15.998915 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:15.998838    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-credential-socket\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.005452 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.005426    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:16.099295 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.099261    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c4xb\" (UniqueName: \"kubernetes.io/projected/b61628be-a3de-4cb7-aba5-d31e11396ad5-kube-api-access-5c4xb\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.099500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.099306    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-data\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.099500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.099330    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-token\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.099500 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.099449    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-podinfo\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.099627 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.099504    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-credential-socket\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.099627 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.099553    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-envoy\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.099627 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.099596    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-workload-certs\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.099816 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.099655    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-workload-socket\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.099816 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.099744    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61628be-a3de-4cb7-aba5-d31e11396ad5-istiod-ca-cert\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.099816 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.099757    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-data\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.100039 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.099843    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-credential-socket\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.100186 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.100159    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-workload-certs\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.100342 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.100193    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-workload-socket\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.100477 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.100454    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61628be-a3de-4cb7-aba5-d31e11396ad5-istiod-ca-cert\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.102111 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.102089    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-envoy\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.102210 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.102151    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-podinfo\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.126084 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.126050    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c4xb\" (UniqueName: \"kubernetes.io/projected/b61628be-a3de-4cb7-aba5-d31e11396ad5-kube-api-access-5c4xb\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.126925 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.126903    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-token\") pod \"e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") " pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.226346 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.226304    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:16.269019 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.268986    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:16.424559 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.424527    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:16.512746 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.512718    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"]
Oct 31 14:10:16.514674 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:10:16.514640    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61628be_a3de_4cb7_aba5_d31e11396ad5.slice/crio-c8450e30f14b517070c59b5e4f1b8334294863a3bba2bd67a66025264382f963 WatchSource:0}: Error finding container c8450e30f14b517070c59b5e4f1b8334294863a3bba2bd67a66025264382f963: Status 404 returned error can't find the container with id c8450e30f14b517070c59b5e4f1b8334294863a3bba2bd67a66025264382f963
Oct 31 14:10:16.516841 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.516807    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:10:16.516955 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.516920    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:10:16.516955 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.516951    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:10:16.944113 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.943993    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4" event={"ID":"b61628be-a3de-4cb7-aba5-d31e11396ad5","Type":"ContainerStarted","Data":"b6877309d89c17ef2cfb41118d5fe386d1a7ea10d0a18d0d5efee2367b311a02"}
Oct 31 14:10:16.944113 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.944062    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4" event={"ID":"b61628be-a3de-4cb7-aba5-d31e11396ad5","Type":"ContainerStarted","Data":"c8450e30f14b517070c59b5e4f1b8334294863a3bba2bd67a66025264382f963"}
Oct 31 14:10:16.971065 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:16.971000    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4" podStartSLOduration=1.9709843839999999 podStartE2EDuration="1.970984384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:10:16.968624099 +0000 UTC m=+4014.366972747" watchObservedRunningTime="2026-10-31 14:10:16.970984384 +0000 UTC m=+4014.369333067"
Oct 31 14:10:17.227217 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:17.227184    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:17.232232 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:17.232204    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:17.950224 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:17.950192    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:17.951239 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:17.951220    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:18.031770 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:18.031731    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"]
Oct 31 14:10:18.035994 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:18.035961    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cn92m"]
Oct 31 14:10:18.740272 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:10:18.740232    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:10:18.753658 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:18.753625    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d060dc2b-6965-4813-9dc4-c9c42c1e9c3c" path="/var/lib/kubelet/pods/d060dc2b-6965-4813-9dc4-c9c42c1e9c3c/volumes"
Oct 31 14:10:19.300663 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:19.300610    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:19.502594 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:19.502564    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:20.442212 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.442167    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"]
Oct 31 14:10:20.446458 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.446427    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:10:20.460284 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.460235    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 14:10:20.460834 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.460795    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bx4t6\""
Oct 31 14:10:20.461171 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.461140    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"]
Oct 31 14:10:20.462360 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.462336    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-c4292c-serving-cert\""
Oct 31 14:10:20.535218 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.535177    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcd3281-61e0-4876-9698-85240b7172d0-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk\" (UID: \"1dcd3281-61e0-4876-9698-85240b7172d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:10:20.535452 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.535326    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dcd3281-61e0-4876-9698-85240b7172d0-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk\" (UID: \"1dcd3281-61e0-4876-9698-85240b7172d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:10:20.535452 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.535441    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-765v9\" (UniqueName: \"kubernetes.io/projected/1dcd3281-61e0-4876-9698-85240b7172d0-kube-api-access-765v9\") pod \"maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk\" (UID: \"1dcd3281-61e0-4876-9698-85240b7172d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:10:20.535584 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.535484    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dcd3281-61e0-4876-9698-85240b7172d0-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk\" (UID: \"1dcd3281-61e0-4876-9698-85240b7172d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:10:20.573247 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.573212    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"]
Oct 31 14:10:20.576991 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.576973    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"
Oct 31 14:10:20.579939 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.579909    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:20.585081 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.585050    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-f13d6eb8-dockercfg-wrh55\""
Oct 31 14:10:20.585216 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.585100    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c4292c\""
Oct 31 14:10:20.588823 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.588795    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"]
Oct 31 14:10:20.624342 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.624312    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"]
Oct 31 14:10:20.628039 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.628016    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"
Oct 31 14:10:20.636431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.636404    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxzlt\" (UniqueName: \"kubernetes.io/projected/5a9df355-090e-4da6-a708-b7acbb252297-kube-api-access-vxzlt\") pod \"payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf\" (UID: \"5a9df355-090e-4da6-a708-b7acbb252297\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"
Oct 31 14:10:20.636664 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.636637    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcd3281-61e0-4876-9698-85240b7172d0-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk\" (UID: \"1dcd3281-61e0-4876-9698-85240b7172d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:10:20.637951 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.636693    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a9df355-090e-4da6-a708-b7acbb252297-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf\" (UID: \"5a9df355-090e-4da6-a708-b7acbb252297\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"
Oct 31 14:10:20.637951 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.636747    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dcd3281-61e0-4876-9698-85240b7172d0-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk\" (UID: \"1dcd3281-61e0-4876-9698-85240b7172d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:10:20.637951 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.636793    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765v9\" (UniqueName: \"kubernetes.io/projected/1dcd3281-61e0-4876-9698-85240b7172d0-kube-api-access-765v9\") pod \"maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk\" (UID: \"1dcd3281-61e0-4876-9698-85240b7172d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:10:20.637951 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.636827    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dcd3281-61e0-4876-9698-85240b7172d0-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk\" (UID: \"1dcd3281-61e0-4876-9698-85240b7172d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:10:20.637951 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.636856    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x9gd\" (UniqueName: \"kubernetes.io/projected/963d02d0-8a8a-493f-8b2c-7b98b17f3413-kube-api-access-6x9gd\") pod \"payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv\" (UID: \"963d02d0-8a8a-493f-8b2c-7b98b17f3413\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"
Oct 31 14:10:20.637951 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.636937    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/963d02d0-8a8a-493f-8b2c-7b98b17f3413-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv\" (UID: \"963d02d0-8a8a-493f-8b2c-7b98b17f3413\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"
Oct 31 14:10:20.637951 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.637507    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcd3281-61e0-4876-9698-85240b7172d0-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk\" (UID: \"1dcd3281-61e0-4876-9698-85240b7172d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:10:20.650431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.640287    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"]
Oct 31 14:10:20.650431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.640600    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dcd3281-61e0-4876-9698-85240b7172d0-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk\" (UID: \"1dcd3281-61e0-4876-9698-85240b7172d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:10:20.650431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.640601    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dcd3281-61e0-4876-9698-85240b7172d0-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk\" (UID: \"1dcd3281-61e0-4876-9698-85240b7172d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:10:20.654334 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.654304    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-765v9\" (UniqueName: \"kubernetes.io/projected/1dcd3281-61e0-4876-9698-85240b7172d0-kube-api-access-765v9\") pod \"maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk\" (UID: \"1dcd3281-61e0-4876-9698-85240b7172d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:10:20.737775 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.737728    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x9gd\" (UniqueName: \"kubernetes.io/projected/963d02d0-8a8a-493f-8b2c-7b98b17f3413-kube-api-access-6x9gd\") pod \"payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv\" (UID: \"963d02d0-8a8a-493f-8b2c-7b98b17f3413\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"
Oct 31 14:10:20.738031 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.737786    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/963d02d0-8a8a-493f-8b2c-7b98b17f3413-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv\" (UID: \"963d02d0-8a8a-493f-8b2c-7b98b17f3413\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"
Oct 31 14:10:20.738031 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.737852    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxzlt\" (UniqueName: \"kubernetes.io/projected/5a9df355-090e-4da6-a708-b7acbb252297-kube-api-access-vxzlt\") pod \"payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf\" (UID: \"5a9df355-090e-4da6-a708-b7acbb252297\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"
Oct 31 14:10:20.738031 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.737933    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a9df355-090e-4da6-a708-b7acbb252297-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf\" (UID: \"5a9df355-090e-4da6-a708-b7acbb252297\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"
Oct 31 14:10:20.738662 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.738642    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/963d02d0-8a8a-493f-8b2c-7b98b17f3413-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv\" (UID: \"963d02d0-8a8a-493f-8b2c-7b98b17f3413\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"
Oct 31 14:10:20.738712 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.738686    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a9df355-090e-4da6-a708-b7acbb252297-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf\" (UID: \"5a9df355-090e-4da6-a708-b7acbb252297\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"
Oct 31 14:10:20.743322 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.743302    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:20.752254 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.752224    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxzlt\" (UniqueName: \"kubernetes.io/projected/5a9df355-090e-4da6-a708-b7acbb252297-kube-api-access-vxzlt\") pod \"payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf\" (UID: \"5a9df355-090e-4da6-a708-b7acbb252297\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"
Oct 31 14:10:20.752395 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.752290    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x9gd\" (UniqueName: \"kubernetes.io/projected/963d02d0-8a8a-493f-8b2c-7b98b17f3413-kube-api-access-6x9gd\") pod \"payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv\" (UID: \"963d02d0-8a8a-493f-8b2c-7b98b17f3413\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"
Oct 31 14:10:20.771521 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.771481    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:10:20.893840 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.893794    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"
Oct 31 14:10:20.946553 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:20.946430    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"
Oct 31 14:10:21.063317 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:21.063274    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"]
Oct 31 14:10:21.065284 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:10:21.065257    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcd3281_61e0_4876_9698_85240b7172d0.slice/crio-108ca279e1359a903820fe6e94bfd5d24877da10a4cc21760e3f9c880a956939 WatchSource:0}: Error finding container 108ca279e1359a903820fe6e94bfd5d24877da10a4cc21760e3f9c880a956939: Status 404 returned error can't find the container with id 108ca279e1359a903820fe6e94bfd5d24877da10a4cc21760e3f9c880a956939
Oct 31 14:10:21.066758 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:21.066740    4428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 14:10:21.174435 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:21.174402    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"]
Oct 31 14:10:21.176115 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:10:21.176071    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9df355_090e_4da6_a708_b7acbb252297.slice/crio-b97f8cda306992bb52f80fd2e5d6a64bb2e52ff1fa8e514294b3444f08b0dfdd WatchSource:0}: Error finding container b97f8cda306992bb52f80fd2e5d6a64bb2e52ff1fa8e514294b3444f08b0dfdd: Status 404 returned error can't find the container with id b97f8cda306992bb52f80fd2e5d6a64bb2e52ff1fa8e514294b3444f08b0dfdd
Oct 31 14:10:21.236050 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:21.236021    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"]
Oct 31 14:10:21.237951 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:10:21.237916    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963d02d0_8a8a_493f_8b2c_7b98b17f3413.slice/crio-7a47690a3edb4a980a24877f154ef8b2456562794818342ef7ab2b1d83c6005b WatchSource:0}: Error finding container 7a47690a3edb4a980a24877f154ef8b2456562794818342ef7ab2b1d83c6005b: Status 404 returned error can't find the container with id 7a47690a3edb4a980a24877f154ef8b2456562794818342ef7ab2b1d83c6005b
Oct 31 14:10:21.559781 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:21.559702    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:21.977057 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:21.977010    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf" event={"ID":"5a9df355-090e-4da6-a708-b7acbb252297","Type":"ContainerStarted","Data":"ad81c708dd8ea546ffc8d7a1362f86b6cc4cc53db18775e9e62028961313b052"}
Oct 31 14:10:21.977253 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:21.977080    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf" event={"ID":"5a9df355-090e-4da6-a708-b7acbb252297","Type":"ContainerStarted","Data":"b97f8cda306992bb52f80fd2e5d6a64bb2e52ff1fa8e514294b3444f08b0dfdd"}
Oct 31 14:10:21.977253 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:21.977242    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"
Oct 31 14:10:21.978676 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:21.978642    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv" event={"ID":"963d02d0-8a8a-493f-8b2c-7b98b17f3413","Type":"ContainerStarted","Data":"d1cc9b6197c08705ad5599147e7d47ef37fdd3a340d364c702aa2ea55d563457"}
Oct 31 14:10:21.978825 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:21.978700    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv" event={"ID":"963d02d0-8a8a-493f-8b2c-7b98b17f3413","Type":"ContainerStarted","Data":"7a47690a3edb4a980a24877f154ef8b2456562794818342ef7ab2b1d83c6005b"}
Oct 31 14:10:21.978825 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:21.978722    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"
Oct 31 14:10:21.980364 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:21.980340    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk" event={"ID":"1dcd3281-61e0-4876-9698-85240b7172d0","Type":"ContainerStarted","Data":"8726bbf52ff7fe7e82ed4eeb45af60e0b5fd62ca9f1eadde7f19c75ae351e420"}
Oct 31 14:10:21.980507 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:21.980379    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk" event={"ID":"1dcd3281-61e0-4876-9698-85240b7172d0","Type":"ContainerStarted","Data":"108ca279e1359a903820fe6e94bfd5d24877da10a4cc21760e3f9c880a956939"}
Oct 31 14:10:21.980507 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:21.980444    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:10:22.000473 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:22.000416    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf" podStartSLOduration=2.000398385 podStartE2EDuration="2.000398385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:10:21.998166099 +0000 UTC m=+4019.396514723" watchObservedRunningTime="2026-10-31 14:10:22.000398385 +0000 UTC m=+4019.398747010"
Oct 31 14:10:22.018815 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:22.018751    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv" podStartSLOduration=2.018739861 podStartE2EDuration="2.018739861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:10:22.017446615 +0000 UTC m=+4019.415795241" watchObservedRunningTime="2026-10-31 14:10:22.018739861 +0000 UTC m=+4019.417088484"
Oct 31 14:10:22.042729 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:22.042665    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk" podStartSLOduration=1.549006348 podStartE2EDuration="2.042644783s" totalImagesPullingTime="493.638435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:10:22.042344085 +0000 UTC m=+4019.440692710" watchObservedRunningTime="2026-10-31 14:10:22.042644783 +0000 UTC m=+4019.440993410"
Oct 31 14:10:22.446319 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:22.446226    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:23.312799 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:23.312765    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:24.139980 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:24.139942    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:24.866892 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:24.866828    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:25.635942 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:25.635897    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:26.445279 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:26.445240    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:27.235730 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:27.235698    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:27.991504 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:27.991471    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:10:28.132598 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:28.132546    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:28.938019 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:28.937988    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:29.781053 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:29.781017    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:30.530089 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:30.530056    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:31.271470 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:31.271439    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:32.050468 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:32.050432    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:32.843862 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:32.843829    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:32.988345 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:32.988311    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"
Oct 31 14:10:32.988580 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:32.988554    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"
Oct 31 14:10:33.368721 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:33.368487    4428 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": read tcp 10.134.0.2:59886->10.134.1.74:15021: read: connection reset by peer" start-of-body=
Oct 31 14:10:33.368721 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:33.368554    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4" podUID="b61628be-a3de-4cb7-aba5-d31e11396ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": read tcp 10.134.0.2:59886->10.134.1.74:15021: read: connection reset by peer"
Oct 31 14:10:33.640744 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:33.640630    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:34.048100 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:34.048030    4428 generic.go:356] "Generic (PLEG): container finished" podID="b61628be-a3de-4cb7-aba5-d31e11396ad5" containerID="b6877309d89c17ef2cfb41118d5fe386d1a7ea10d0a18d0d5efee2367b311a02" exitCode=0
Oct 31 14:10:34.048921 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:34.048744    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4" event={"ID":"b61628be-a3de-4cb7-aba5-d31e11396ad5","Type":"ContainerDied","Data":"b6877309d89c17ef2cfb41118d5fe386d1a7ea10d0a18d0d5efee2367b311a02"}
Oct 31 14:10:34.049479 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:34.049451    4428 scope.go:122] "RemoveContainer" containerID="b6877309d89c17ef2cfb41118d5fe386d1a7ea10d0a18d0d5efee2367b311a02"
Oct 31 14:10:34.050749 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:34.050721    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:10:34.050812 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:34.050797    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:10:34.050850 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:34.050834    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:10:34.363951 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:34.363817    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:34.524648 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:34.524613    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:35.010342 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:35.010306    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:35.076957 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:35.076922    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4" event={"ID":"b61628be-a3de-4cb7-aba5-d31e11396ad5","Type":"ContainerStarted","Data":"edab17b5b904d2eb232365fa0836f0318ad8b381022beff36d02c129a47ce317"}
Oct 31 14:10:35.192150 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:35.192113    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:35.227306 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:35.227281    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:35.227577 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:35.227545    4428 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body=
Oct 31 14:10:35.227689 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:35.227600    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4" podUID="b61628be-a3de-4cb7-aba5-d31e11396ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused"
Oct 31 14:10:35.379688 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:35.379600    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:36.227130 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.227090    4428 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body=
Oct 31 14:10:36.227596 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.227161    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4" podUID="b61628be-a3de-4cb7-aba5-d31e11396ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused"
Oct 31 14:10:36.231305 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.231274    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:36.688684 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.688581    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"]
Oct 31 14:10:36.692701 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.692674    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.698387 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.698357    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c4292c\"/\"kube-root-ca.crt\""
Oct 31 14:10:36.699120 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.698920    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c4292c\"/\"openshift-service-ca.crt\""
Oct 31 14:10:36.700055 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.700034    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c4292c\"/\"default-dockercfg-qk74g\""
Oct 31 14:10:36.701030 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.701010    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c4292c\"/\"simulator-single-main-c4292c-kserve-self-signed-certs\""
Oct 31 14:10:36.705126 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.705098    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"]
Oct 31 14:10:36.805306 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.805269    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-dshm\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.805499 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.805324    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss7bv\" (UniqueName: \"kubernetes.io/projected/3bee5357-8b0a-4dd9-af2f-848c546f7d66-kube-api-access-ss7bv\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.805499 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.805382    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-tmp-dir\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.805499 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.805464    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-home\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.805625 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.805520    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-model-cache\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.805625 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.805537    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bee5357-8b0a-4dd9-af2f-848c546f7d66-tls-certs\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.822994 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.822960    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:36.906657 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.906619    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-dshm\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.906657 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.906663    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss7bv\" (UniqueName: \"kubernetes.io/projected/3bee5357-8b0a-4dd9-af2f-848c546f7d66-kube-api-access-ss7bv\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.907008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.906723    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-tmp-dir\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.907008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.906752    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-home\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.907008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.906787    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-model-cache\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.907008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.906809    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bee5357-8b0a-4dd9-af2f-848c546f7d66-tls-certs\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.907365 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.907340    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-tmp-dir\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.908071 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.907964    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-home\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.908071 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.908038    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-model-cache\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.911175 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.911149    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bee5357-8b0a-4dd9-af2f-848c546f7d66-tls-certs\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.911359 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.911333    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-dshm\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:36.920999 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:36.920969    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss7bv\" (UniqueName: \"kubernetes.io/projected/3bee5357-8b0a-4dd9-af2f-848c546f7d66-kube-api-access-ss7bv\") pod \"simulator-single-main-c4292c-kserve-85db47f67f-5t6k9\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") " pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:37.016129 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:37.016083    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:37.178845 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:37.178808    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:37.227021 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:37.226986    4428 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body=
Oct 31 14:10:37.227198 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:37.227042    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4" podUID="b61628be-a3de-4cb7-aba5-d31e11396ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused"
Oct 31 14:10:37.321182 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:37.321154    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"]
Oct 31 14:10:37.322786 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:10:37.322760    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bee5357_8b0a_4dd9_af2f_848c546f7d66.slice/crio-4877136164a4c265cb8250d98417e8cacc4233aa7f84e88a790af5c54ed19e0f WatchSource:0}: Error finding container 4877136164a4c265cb8250d98417e8cacc4233aa7f84e88a790af5c54ed19e0f: Status 404 returned error can't find the container with id 4877136164a4c265cb8250d98417e8cacc4233aa7f84e88a790af5c54ed19e0f
Oct 31 14:10:38.043008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:38.042973    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:38.095812 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:38.095770    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9" event={"ID":"3bee5357-8b0a-4dd9-af2f-848c546f7d66","Type":"ContainerStarted","Data":"c253ad9c7776739c89f5ff3fca3171702568ddfadc98bd8b234c85fe322398ed"}
Oct 31 14:10:38.096032 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:38.095821    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9" event={"ID":"3bee5357-8b0a-4dd9-af2f-848c546f7d66","Type":"ContainerStarted","Data":"4877136164a4c265cb8250d98417e8cacc4233aa7f84e88a790af5c54ed19e0f"}
Oct 31 14:10:38.122238 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:38.122171    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9" podStartSLOduration=2.122150993 podStartE2EDuration="2.122150993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:10:38.121487339 +0000 UTC m=+4035.519835963" watchObservedRunningTime="2026-10-31 14:10:38.122150993 +0000 UTC m=+4035.520499618"
Oct 31 14:10:38.227094 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:38.227058    4428 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body=
Oct 31 14:10:38.227297 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:38.227112    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4" podUID="b61628be-a3de-4cb7-aba5-d31e11396ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused"
Oct 31 14:10:38.969899 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:38.969533    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:39.016949 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:39.016906    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:39.227055 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:39.226953    4428 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body=
Oct 31 14:10:39.227055 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:39.227023    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4" podUID="b61628be-a3de-4cb7-aba5-d31e11396ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused"
Oct 31 14:10:39.839728 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:39.839679    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:40.617628 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:40.617589    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:40.618081 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:40.618050    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:40.619089 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:40.619054    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:10:40.636990 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:40.636949    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:41.516956 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:41.516923    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:42.390895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:42.390830    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:43.159112 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:43.159072    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:44.008388 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:44.008348    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:44.825800 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:44.825764    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:45.550903 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:45.550847    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:46.342062 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:46.342030    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:47.016687 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:47.016652    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:47.029718 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:47.029676    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:47.156581 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:47.156550    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:10:47.166094 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:47.166062    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:47.954725 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:47.954689    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:48.743137 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:48.743101    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:48.745061 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:10:48.745028    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:10:49.735734 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:49.735699    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:50.537934 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:50.537900    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:51.377407 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:51.377371    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:52.279025 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:52.278992    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:53.143940 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:53.143906    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:53.943839 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:53.943806    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:54.169780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:54.169748    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:54.812746 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:54.812687    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:55.746854 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:55.746811    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:56.545924 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:56.545884    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:57.352289 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:57.352249    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:58.252825 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:58.252780    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:10:59.232997 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:10:59.232956    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:00.034047 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:00.034002    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:00.857934 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:00.857890    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:01.654298 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:01.654262    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:02.552742 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:02.552708    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:03.337805 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:03.337766    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:04.232497 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:04.232462    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:05.034401 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:05.034356    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:05.850025 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:05.849983    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:06.647803 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:06.647749    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:07.481667 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:07.481631    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:08.246689 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:08.246642    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:09.041276 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:09.041234    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:09.843459 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:09.843410    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:10.044679 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:10.044643    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"]
Oct 31 14:11:10.051148 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:10.050683    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tvpjd"]
Oct 31 14:11:10.737468 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:10.737433    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:10.753774 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:10.753737    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a9c61f-10c8-449f-9dd1-39b13c846c5e" path="/var/lib/kubelet/pods/47a9c61f-10c8-449f-9dd1-39b13c846c5e/volumes"
Oct 31 14:11:11.648772 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:11.648735    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:12.446292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:12.446247    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:13.251404 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:13.251371    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:14.091266 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:14.091217    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:15.968591 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:15.968555    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:16.135727 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:16.135681    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:16.287330 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:16.287227    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:16.537083 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:16.537044    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:17.382213 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:17.382178    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:17.540797 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:17.540759    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:18.403199 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:18.403157    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:18.744062 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:11:18.744027    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:11:19.345451 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:19.345419    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:20.138917 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:20.138859    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:20.936492 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:20.936446    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:21.739398 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:21.739364    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:22.265417 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:22.265382    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"]
Oct 31 14:11:22.424814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:22.424771    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:22.593913 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:22.593792    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:23.349231 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:23.349195    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:24.215893 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:24.215830    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:25.030491 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:25.030453    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:25.840066 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:25.840015    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:26.707505 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:26.707449    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:27.497057 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:27.497025    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:27.524013 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:27.523973    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"]
Oct 31 14:11:27.551948 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:27.551910    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"]
Oct 31 14:11:27.678445 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:27.678413    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:27.842849 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:27.842767    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:28.335248 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:28.335218    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:28.868394 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:28.868359    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"]
Oct 31 14:11:28.868971 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:28.868590    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk" podUID="1dcd3281-61e0-4876-9698-85240b7172d0" containerName="maas-api" containerID="cri-o://8726bbf52ff7fe7e82ed4eeb45af60e0b5fd62ca9f1eadde7f19c75ae351e420" gracePeriod=30
Oct 31 14:11:28.975428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:28.975394    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:29.136926 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.136899    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:11:29.146236 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.146203    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:29.237880 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.237829    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dcd3281-61e0-4876-9698-85240b7172d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcd3281-61e0-4876-9698-85240b7172d0-openshift-service-ca\") pod \"1dcd3281-61e0-4876-9698-85240b7172d0\" (UID: \"1dcd3281-61e0-4876-9698-85240b7172d0\") "
Oct 31 14:11:29.238105 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.237896    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dcd3281-61e0-4876-9698-85240b7172d0-kube-api-access-765v9\" (UniqueName: \"kubernetes.io/projected/1dcd3281-61e0-4876-9698-85240b7172d0-kube-api-access-765v9\") pod \"1dcd3281-61e0-4876-9698-85240b7172d0\" (UID: \"1dcd3281-61e0-4876-9698-85240b7172d0\") "
Oct 31 14:11:29.238105 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.238025    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dcd3281-61e0-4876-9698-85240b7172d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dcd3281-61e0-4876-9698-85240b7172d0-metrics-certs\") pod \"1dcd3281-61e0-4876-9698-85240b7172d0\" (UID: \"1dcd3281-61e0-4876-9698-85240b7172d0\") "
Oct 31 14:11:29.238105 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.238057    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dcd3281-61e0-4876-9698-85240b7172d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dcd3281-61e0-4876-9698-85240b7172d0-maas-api-tls\") pod \"1dcd3281-61e0-4876-9698-85240b7172d0\" (UID: \"1dcd3281-61e0-4876-9698-85240b7172d0\") "
Oct 31 14:11:29.238333 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.238302    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dcd3281-61e0-4876-9698-85240b7172d0-openshift-service-ca" pod "1dcd3281-61e0-4876-9698-85240b7172d0" (UID: "1dcd3281-61e0-4876-9698-85240b7172d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:11:29.238440 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.238422    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcd3281-61e0-4876-9698-85240b7172d0-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:29.240249 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.240226    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dcd3281-61e0-4876-9698-85240b7172d0-maas-api-tls" pod "1dcd3281-61e0-4876-9698-85240b7172d0" (UID: "1dcd3281-61e0-4876-9698-85240b7172d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:11:29.240340 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.240246    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dcd3281-61e0-4876-9698-85240b7172d0-metrics-certs" pod "1dcd3281-61e0-4876-9698-85240b7172d0" (UID: "1dcd3281-61e0-4876-9698-85240b7172d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:11:29.240378 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.240351    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dcd3281-61e0-4876-9698-85240b7172d0-kube-api-access-765v9" pod "1dcd3281-61e0-4876-9698-85240b7172d0" (UID: "1dcd3281-61e0-4876-9698-85240b7172d0"). InnerVolumeSpecName "kube-api-access-765v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:11:29.305450 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.305416    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:29.339682 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.339641    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-765v9\" (UniqueName: \"kubernetes.io/projected/1dcd3281-61e0-4876-9698-85240b7172d0-kube-api-access-765v9\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:29.339682 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.339676    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dcd3281-61e0-4876-9698-85240b7172d0-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:29.339682 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.339686    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dcd3281-61e0-4876-9698-85240b7172d0-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:29.373033 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.372944    4428 generic.go:356] "Generic (PLEG): container finished" podID="1dcd3281-61e0-4876-9698-85240b7172d0" containerID="8726bbf52ff7fe7e82ed4eeb45af60e0b5fd62ca9f1eadde7f19c75ae351e420" exitCode=0
Oct 31 14:11:29.373033 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.373002    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"
Oct 31 14:11:29.373252 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.373035    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk" event={"ID":"1dcd3281-61e0-4876-9698-85240b7172d0","Type":"ContainerDied","Data":"8726bbf52ff7fe7e82ed4eeb45af60e0b5fd62ca9f1eadde7f19c75ae351e420"}
Oct 31 14:11:29.373252 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.373096    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk" event={"ID":"1dcd3281-61e0-4876-9698-85240b7172d0","Type":"ContainerDied","Data":"108ca279e1359a903820fe6e94bfd5d24877da10a4cc21760e3f9c880a956939"}
Oct 31 14:11:29.373252 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.373119    4428 scope.go:122] "RemoveContainer" containerID="8726bbf52ff7fe7e82ed4eeb45af60e0b5fd62ca9f1eadde7f19c75ae351e420"
Oct 31 14:11:29.383861 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.383842    4428 scope.go:122] "RemoveContainer" containerID="8726bbf52ff7fe7e82ed4eeb45af60e0b5fd62ca9f1eadde7f19c75ae351e420"
Oct 31 14:11:29.384193 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:11:29.384174    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8726bbf52ff7fe7e82ed4eeb45af60e0b5fd62ca9f1eadde7f19c75ae351e420\": container with ID starting with 8726bbf52ff7fe7e82ed4eeb45af60e0b5fd62ca9f1eadde7f19c75ae351e420 not found: ID does not exist" containerID="8726bbf52ff7fe7e82ed4eeb45af60e0b5fd62ca9f1eadde7f19c75ae351e420"
Oct 31 14:11:29.384279 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.384200    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8726bbf52ff7fe7e82ed4eeb45af60e0b5fd62ca9f1eadde7f19c75ae351e420"} err="failed to get container status \"8726bbf52ff7fe7e82ed4eeb45af60e0b5fd62ca9f1eadde7f19c75ae351e420\": rpc error: code = NotFound desc = could not find container \"8726bbf52ff7fe7e82ed4eeb45af60e0b5fd62ca9f1eadde7f19c75ae351e420\": container with ID starting with 8726bbf52ff7fe7e82ed4eeb45af60e0b5fd62ca9f1eadde7f19c75ae351e420 not found: ID does not exist"
Oct 31 14:11:29.405033 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.405003    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"]
Oct 31 14:11:29.410300 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.410272    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4292c-65f89bd6f5-lf8rk"]
Oct 31 14:11:29.721208 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:29.721174    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:30.754747 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:30.754712    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dcd3281-61e0-4876-9698-85240b7172d0" path="/var/lib/kubelet/pods/1dcd3281-61e0-4876-9698-85240b7172d0/volumes"
Oct 31 14:11:31.029511 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:31.029420    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"]
Oct 31 14:11:31.036050 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:31.036017    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vdzb7"]
Oct 31 14:11:32.524852 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:32.524784    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv" podUID="963d02d0-8a8a-493f-8b2c-7b98b17f3413" containerName="payload-processing" containerID="cri-o://d1cc9b6197c08705ad5599147e7d47ef37fdd3a340d364c702aa2ea55d563457" gracePeriod=25
Oct 31 14:11:32.553042 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:32.552981    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf" podUID="5a9df355-090e-4da6-a708-b7acbb252297" containerName="payload-pre-processing" containerID="cri-o://ad81c708dd8ea546ffc8d7a1362f86b6cc4cc53db18775e9e62028961313b052" gracePeriod=25
Oct 31 14:11:32.755709 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:32.755673    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def6c960-6bfa-4b73-986b-8b525f3baa9f" path="/var/lib/kubelet/pods/def6c960-6bfa-4b73-986b-8b525f3baa9f/volumes"
Oct 31 14:11:32.986802 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:32.986763    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf" podUID="5a9df355-090e-4da6-a708-b7acbb252297" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:11:32.986802 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:32.986784    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv" podUID="963d02d0-8a8a-493f-8b2c-7b98b17f3413" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:11:36.130813 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:36.130705    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:36.225733 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:36.225695    4428 scope.go:122] "RemoveContainer" containerID="d570083bcc6bfb40e6bd93fc81826f7f174949e9b756ace73ee0bd4af6698533"
Oct 31 14:11:36.293905 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:36.293852    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:36.409703 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:36.409353    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"]
Oct 31 14:11:36.410325 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:36.410285    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4" podUID="b61628be-a3de-4cb7-aba5-d31e11396ad5" containerName="istio-proxy" containerID="cri-o://edab17b5b904d2eb232365fa0836f0318ad8b381022beff36d02c129a47ce317" gracePeriod=30
Oct 31 14:11:36.506076 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:36.506041    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:11:37.286326 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.286254    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9" podUID="3bee5357-8b0a-4dd9-af2f-848c546f7d66" containerName="main" containerID="cri-o://c253ad9c7776739c89f5ff3fca3171702568ddfadc98bd8b234c85fe322398ed" gracePeriod=45
Oct 31 14:11:37.440393 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.440342    4428 generic.go:356] "Generic (PLEG): container finished" podID="3bee5357-8b0a-4dd9-af2f-848c546f7d66" containerID="c253ad9c7776739c89f5ff3fca3171702568ddfadc98bd8b234c85fe322398ed" exitCode=0
Oct 31 14:11:37.440590 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.440411    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9" event={"ID":"3bee5357-8b0a-4dd9-af2f-848c546f7d66","Type":"ContainerDied","Data":"c253ad9c7776739c89f5ff3fca3171702568ddfadc98bd8b234c85fe322398ed"}
Oct 31 14:11:37.548346 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.548270    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:11:37.643175 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.643141    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-dshm\") pod \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") "
Oct 31 14:11:37.643386 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.643190    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bee5357-8b0a-4dd9-af2f-848c546f7d66-tls-certs\" (UniqueName: \"kubernetes.io/secret/3bee5357-8b0a-4dd9-af2f-848c546f7d66-tls-certs\") pod \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") "
Oct 31 14:11:37.643386 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.643213    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bee5357-8b0a-4dd9-af2f-848c546f7d66-kube-api-access-ss7bv\" (UniqueName: \"kubernetes.io/projected/3bee5357-8b0a-4dd9-af2f-848c546f7d66-kube-api-access-ss7bv\") pod \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") "
Oct 31 14:11:37.643386 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.643238    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-model-cache\") pod \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") "
Oct 31 14:11:37.643386 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.643308    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-home\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-home\") pod \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") "
Oct 31 14:11:37.643386 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.643382    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-tmp-dir\") pod \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\" (UID: \"3bee5357-8b0a-4dd9-af2f-848c546f7d66\") "
Oct 31 14:11:37.643707 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.643656    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-model-cache" pod "3bee5357-8b0a-4dd9-af2f-848c546f7d66" (UID: "3bee5357-8b0a-4dd9-af2f-848c546f7d66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:11:37.643905 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.643823    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:37.643969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.643921    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-tmp-dir" pod "3bee5357-8b0a-4dd9-af2f-848c546f7d66" (UID: "3bee5357-8b0a-4dd9-af2f-848c546f7d66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:11:37.644149 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.644123    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-home" pod "3bee5357-8b0a-4dd9-af2f-848c546f7d66" (UID: "3bee5357-8b0a-4dd9-af2f-848c546f7d66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:11:37.646017 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.645955    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bee5357-8b0a-4dd9-af2f-848c546f7d66-tls-certs" pod "3bee5357-8b0a-4dd9-af2f-848c546f7d66" (UID: "3bee5357-8b0a-4dd9-af2f-848c546f7d66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:11:37.646017 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.645975    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bee5357-8b0a-4dd9-af2f-848c546f7d66-kube-api-access-ss7bv" pod "3bee5357-8b0a-4dd9-af2f-848c546f7d66" (UID: "3bee5357-8b0a-4dd9-af2f-848c546f7d66"). InnerVolumeSpecName "kube-api-access-ss7bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:11:37.646230 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.646211    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-dshm" pod "3bee5357-8b0a-4dd9-af2f-848c546f7d66" (UID: "3bee5357-8b0a-4dd9-af2f-848c546f7d66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:11:37.699109 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.699073    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"]
Oct 31 14:11:37.708261 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.708227    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hqvl4"]
Oct 31 14:11:37.736658 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.736621    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"]
Oct 31 14:11:37.737107 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.737093    4428 cpu_manager.go:414] "containerMap: removing container" podUID="1dcd3281-61e0-4876-9698-85240b7172d0" containerName="maas-api"
Oct 31 14:11:37.737176 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.737108    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dcd3281-61e0-4876-9698-85240b7172d0" containerName="maas-api"
Oct 31 14:11:37.737176 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.737133    4428 cpu_manager.go:414] "containerMap: removing container" podUID="3bee5357-8b0a-4dd9-af2f-848c546f7d66" containerName="main"
Oct 31 14:11:37.737176 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.737139    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bee5357-8b0a-4dd9-af2f-848c546f7d66" containerName="main"
Oct 31 14:11:37.737311 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.737194    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bee5357-8b0a-4dd9-af2f-848c546f7d66" containerName="main"
Oct 31 14:11:37.737311 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.737206    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dcd3281-61e0-4876-9698-85240b7172d0" containerName="maas-api"
Oct 31 14:11:37.740349 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.740333    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"
Oct 31 14:11:37.745090 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.745065    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:37.745090 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.745089    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:37.745261 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.745098    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bee5357-8b0a-4dd9-af2f-848c546f7d66-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:37.745261 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.745108    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss7bv\" (UniqueName: \"kubernetes.io/projected/3bee5357-8b0a-4dd9-af2f-848c546f7d66-kube-api-access-ss7bv\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:37.745261 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.745120    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bee5357-8b0a-4dd9-af2f-848c546f7d66-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:37.748315 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.747754    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:11:37.748939 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.748917    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:11:37.750731 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.750677    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"]
Oct 31 14:11:37.845820 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.845720    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a239bc40-aa03-40ae-b7ff-c33ef400f867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgzbt\" (UID: \"a239bc40-aa03-40ae-b7ff-c33ef400f867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"
Oct 31 14:11:37.845820 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.845775    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh7qn\" (UniqueName: \"kubernetes.io/projected/a239bc40-aa03-40ae-b7ff-c33ef400f867-kube-api-access-vh7qn\") pod \"maas-api-revoke-sub-3d4156f1-cgzbt\" (UID: \"a239bc40-aa03-40ae-b7ff-c33ef400f867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"
Oct 31 14:11:37.846078 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.845932    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a239bc40-aa03-40ae-b7ff-c33ef400f867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgzbt\" (UID: \"a239bc40-aa03-40ae-b7ff-c33ef400f867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"
Oct 31 14:11:37.947189 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.947152    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a239bc40-aa03-40ae-b7ff-c33ef400f867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgzbt\" (UID: \"a239bc40-aa03-40ae-b7ff-c33ef400f867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"
Oct 31 14:11:37.947189 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.947195    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7qn\" (UniqueName: \"kubernetes.io/projected/a239bc40-aa03-40ae-b7ff-c33ef400f867-kube-api-access-vh7qn\") pod \"maas-api-revoke-sub-3d4156f1-cgzbt\" (UID: \"a239bc40-aa03-40ae-b7ff-c33ef400f867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"
Oct 31 14:11:37.947489 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.947228    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a239bc40-aa03-40ae-b7ff-c33ef400f867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgzbt\" (UID: \"a239bc40-aa03-40ae-b7ff-c33ef400f867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"
Oct 31 14:11:37.947954 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.947857    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a239bc40-aa03-40ae-b7ff-c33ef400f867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgzbt\" (UID: \"a239bc40-aa03-40ae-b7ff-c33ef400f867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"
Oct 31 14:11:37.949785 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.949724    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a239bc40-aa03-40ae-b7ff-c33ef400f867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgzbt\" (UID: \"a239bc40-aa03-40ae-b7ff-c33ef400f867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"
Oct 31 14:11:37.960998 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:37.960973    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh7qn\" (UniqueName: \"kubernetes.io/projected/a239bc40-aa03-40ae-b7ff-c33ef400f867-kube-api-access-vh7qn\") pod \"maas-api-revoke-sub-3d4156f1-cgzbt\" (UID: \"a239bc40-aa03-40ae-b7ff-c33ef400f867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"
Oct 31 14:11:38.056982 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:38.056940    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"
Oct 31 14:11:38.337988 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:38.337952    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"]
Oct 31 14:11:38.339804 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:11:38.339767    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda239bc40_aa03_40ae_b7ff_c33ef400f867.slice/crio-47834e127ce53dbdb2ebc2d12cac44e0d6fa0348534539aa7c82ae8d486d6aaf WatchSource:0}: Error finding container 47834e127ce53dbdb2ebc2d12cac44e0d6fa0348534539aa7c82ae8d486d6aaf: Status 404 returned error can't find the container with id 47834e127ce53dbdb2ebc2d12cac44e0d6fa0348534539aa7c82ae8d486d6aaf
Oct 31 14:11:38.447048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:38.447010    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt" event={"ID":"a239bc40-aa03-40ae-b7ff-c33ef400f867","Type":"ContainerStarted","Data":"47834e127ce53dbdb2ebc2d12cac44e0d6fa0348534539aa7c82ae8d486d6aaf"}
Oct 31 14:11:38.448603 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:38.448575    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"
Oct 31 14:11:38.448744 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:38.448610    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9" event={"ID":"3bee5357-8b0a-4dd9-af2f-848c546f7d66","Type":"ContainerDied","Data":"4877136164a4c265cb8250d98417e8cacc4233aa7f84e88a790af5c54ed19e0f"}
Oct 31 14:11:38.448744 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:38.448656    4428 scope.go:122] "RemoveContainer" containerID="c253ad9c7776739c89f5ff3fca3171702568ddfadc98bd8b234c85fe322398ed"
Oct 31 14:11:38.478772 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:38.478737    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"]
Oct 31 14:11:38.485750 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:38.485724    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-c4292c/simulator-single-main-c4292c-kserve-85db47f67f-5t6k9"]
Oct 31 14:11:38.754066 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:38.754035    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bee5357-8b0a-4dd9-af2f-848c546f7d66" path="/var/lib/kubelet/pods/3bee5357-8b0a-4dd9-af2f-848c546f7d66/volumes"
Oct 31 14:11:38.754690 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:38.754490    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4" path="/var/lib/kubelet/pods/fcaa2b41-be05-4d2e-9b6d-ec3814a2b2e4/volumes"
Oct 31 14:11:39.455770 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:39.455738    4428 generic.go:356] "Generic (PLEG): container finished" podID="a239bc40-aa03-40ae-b7ff-c33ef400f867" containerID="479f9eb32b4578887ef9a5fdc053b80ccd248b69f8e1fa7a8b0377f7db6a95a5" exitCode=0
Oct 31 14:11:39.456359 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:39.455825    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt" event={"ID":"a239bc40-aa03-40ae-b7ff-c33ef400f867","Type":"ContainerDied","Data":"479f9eb32b4578887ef9a5fdc053b80ccd248b69f8e1fa7a8b0377f7db6a95a5"}
Oct 31 14:11:40.606625 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:40.606599    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"
Oct 31 14:11:40.620544 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:40.620512    4428 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:11:40.620713 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:40.620565    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4" podUID="b61628be-a3de-4cb7-aba5-d31e11396ad5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:11:40.674858 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:40.674823    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a239bc40-aa03-40ae-b7ff-c33ef400f867-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a239bc40-aa03-40ae-b7ff-c33ef400f867-maas-api-xks-ca\") pod \"a239bc40-aa03-40ae-b7ff-c33ef400f867\" (UID: \"a239bc40-aa03-40ae-b7ff-c33ef400f867\") "
Oct 31 14:11:40.675106 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:40.674906    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a239bc40-aa03-40ae-b7ff-c33ef400f867-kube-api-access-vh7qn\" (UniqueName: \"kubernetes.io/projected/a239bc40-aa03-40ae-b7ff-c33ef400f867-kube-api-access-vh7qn\") pod \"a239bc40-aa03-40ae-b7ff-c33ef400f867\" (UID: \"a239bc40-aa03-40ae-b7ff-c33ef400f867\") "
Oct 31 14:11:40.675106 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:40.674965    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a239bc40-aa03-40ae-b7ff-c33ef400f867-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a239bc40-aa03-40ae-b7ff-c33ef400f867-maas-api-openshift-ca\") pod \"a239bc40-aa03-40ae-b7ff-c33ef400f867\" (UID: \"a239bc40-aa03-40ae-b7ff-c33ef400f867\") "
Oct 31 14:11:40.675379 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:40.675353    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a239bc40-aa03-40ae-b7ff-c33ef400f867-maas-api-openshift-ca" pod "a239bc40-aa03-40ae-b7ff-c33ef400f867" (UID: "a239bc40-aa03-40ae-b7ff-c33ef400f867"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:11:40.677233 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:40.677207    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a239bc40-aa03-40ae-b7ff-c33ef400f867-maas-api-xks-ca" pod "a239bc40-aa03-40ae-b7ff-c33ef400f867" (UID: "a239bc40-aa03-40ae-b7ff-c33ef400f867"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:11:40.677350 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:40.677234    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a239bc40-aa03-40ae-b7ff-c33ef400f867-kube-api-access-vh7qn" pod "a239bc40-aa03-40ae-b7ff-c33ef400f867" (UID: "a239bc40-aa03-40ae-b7ff-c33ef400f867"). InnerVolumeSpecName "kube-api-access-vh7qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:11:40.775662 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:40.775632    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a239bc40-aa03-40ae-b7ff-c33ef400f867-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:40.775662 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:40.775660    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh7qn\" (UniqueName: \"kubernetes.io/projected/a239bc40-aa03-40ae-b7ff-c33ef400f867-kube-api-access-vh7qn\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:40.775911 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:40.775672    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a239bc40-aa03-40ae-b7ff-c33ef400f867-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:41.470117 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:41.470066    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt" event={"ID":"a239bc40-aa03-40ae-b7ff-c33ef400f867","Type":"ContainerDied","Data":"47834e127ce53dbdb2ebc2d12cac44e0d6fa0348534539aa7c82ae8d486d6aaf"}
Oct 31 14:11:41.470335 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:41.470130    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"
Oct 31 14:11:41.470335 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:41.470174    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47834e127ce53dbdb2ebc2d12cac44e0d6fa0348534539aa7c82ae8d486d6aaf"
Oct 31 14:11:41.473063 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:41.473030    4428 generic.go:356] "Generic (PLEG): container finished" podID="b61628be-a3de-4cb7-aba5-d31e11396ad5" containerID="edab17b5b904d2eb232365fa0836f0318ad8b381022beff36d02c129a47ce317" exitCode=0
Oct 31 14:11:41.473232 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:41.473212    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4" event={"ID":"b61628be-a3de-4cb7-aba5-d31e11396ad5","Type":"ContainerDied","Data":"edab17b5b904d2eb232365fa0836f0318ad8b381022beff36d02c129a47ce317"}
Oct 31 14:11:41.473307 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:41.473291    4428 scope.go:122] "RemoveContainer" containerID="b6877309d89c17ef2cfb41118d5fe386d1a7ea10d0a18d0d5efee2367b311a02"
Oct 31 14:11:42.041125 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.041051    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"]
Oct 31 14:11:42.044198 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.044154    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xt47v"]
Oct 31 14:11:42.075219 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.075191    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:11:42.085886 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.085839    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-podinfo\") pod \"b61628be-a3de-4cb7-aba5-d31e11396ad5\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") "
Oct 31 14:11:42.086026 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.085949    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-envoy\") pod \"b61628be-a3de-4cb7-aba5-d31e11396ad5\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") "
Oct 31 14:11:42.086026 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.085990    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-credential-socket\") pod \"b61628be-a3de-4cb7-aba5-d31e11396ad5\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") "
Oct 31 14:11:42.086138 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.086058    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-workload-socket\") pod \"b61628be-a3de-4cb7-aba5-d31e11396ad5\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") "
Oct 31 14:11:42.086138 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.086102    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-workload-certs\") pod \"b61628be-a3de-4cb7-aba5-d31e11396ad5\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") "
Oct 31 14:11:42.086261 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.086141    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61628be-a3de-4cb7-aba5-d31e11396ad5-kube-api-access-5c4xb\" (UniqueName: \"kubernetes.io/projected/b61628be-a3de-4cb7-aba5-d31e11396ad5-kube-api-access-5c4xb\") pod \"b61628be-a3de-4cb7-aba5-d31e11396ad5\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") "
Oct 31 14:11:42.086261 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.086182    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-data\") pod \"b61628be-a3de-4cb7-aba5-d31e11396ad5\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") "
Oct 31 14:11:42.086376 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.086312    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-credential-socket" pod "b61628be-a3de-4cb7-aba5-d31e11396ad5" (UID: "b61628be-a3de-4cb7-aba5-d31e11396ad5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:11:42.086376 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.086327    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-workload-socket" pod "b61628be-a3de-4cb7-aba5-d31e11396ad5" (UID: "b61628be-a3de-4cb7-aba5-d31e11396ad5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:11:42.086498 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.086448    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-workload-certs" pod "b61628be-a3de-4cb7-aba5-d31e11396ad5" (UID: "b61628be-a3de-4cb7-aba5-d31e11396ad5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:11:42.086610 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.086580    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-data" pod "b61628be-a3de-4cb7-aba5-d31e11396ad5" (UID: "b61628be-a3de-4cb7-aba5-d31e11396ad5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:11:42.086699 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.086611    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:42.086699 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.086632    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:42.086699 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.086647    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:42.088463 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.088441    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-envoy" pod "b61628be-a3de-4cb7-aba5-d31e11396ad5" (UID: "b61628be-a3de-4cb7-aba5-d31e11396ad5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:11:42.088603 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.088499    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-podinfo" pod "b61628be-a3de-4cb7-aba5-d31e11396ad5" (UID: "b61628be-a3de-4cb7-aba5-d31e11396ad5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:11:42.088784 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.088762    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61628be-a3de-4cb7-aba5-d31e11396ad5-kube-api-access-5c4xb" pod "b61628be-a3de-4cb7-aba5-d31e11396ad5" (UID: "b61628be-a3de-4cb7-aba5-d31e11396ad5"). InnerVolumeSpecName "kube-api-access-5c4xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:11:42.187784 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.187693    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-token\" (UniqueName: \"kubernetes.io/projected/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-token\") pod \"b61628be-a3de-4cb7-aba5-d31e11396ad5\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") "
Oct 31 14:11:42.187784 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.187734    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61628be-a3de-4cb7-aba5-d31e11396ad5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61628be-a3de-4cb7-aba5-d31e11396ad5-istiod-ca-cert\") pod \"b61628be-a3de-4cb7-aba5-d31e11396ad5\" (UID: \"b61628be-a3de-4cb7-aba5-d31e11396ad5\") "
Oct 31 14:11:42.188039 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.188003    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c4xb\" (UniqueName: \"kubernetes.io/projected/b61628be-a3de-4cb7-aba5-d31e11396ad5-kube-api-access-5c4xb\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:42.188039 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.188015    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:42.188039 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.188024    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:42.188039 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.188033    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:42.188191 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.188161    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61628be-a3de-4cb7-aba5-d31e11396ad5-istiod-ca-cert" pod "b61628be-a3de-4cb7-aba5-d31e11396ad5" (UID: "b61628be-a3de-4cb7-aba5-d31e11396ad5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:11:42.190179 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.190157    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-token" pod "b61628be-a3de-4cb7-aba5-d31e11396ad5" (UID: "b61628be-a3de-4cb7-aba5-d31e11396ad5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:11:42.288694 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.288636    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61628be-a3de-4cb7-aba5-d31e11396ad5-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:42.288694 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.288690    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61628be-a3de-4cb7-aba5-d31e11396ad5-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:42.479988 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.479955    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"
Oct 31 14:11:42.480217 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.479992    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4" event={"ID":"b61628be-a3de-4cb7-aba5-d31e11396ad5","Type":"ContainerDied","Data":"c8450e30f14b517070c59b5e4f1b8334294863a3bba2bd67a66025264382f963"}
Oct 31 14:11:42.480217 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.480059    4428 scope.go:122] "RemoveContainer" containerID="edab17b5b904d2eb232365fa0836f0318ad8b381022beff36d02c129a47ce317"
Oct 31 14:11:42.507634 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.507599    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"]
Oct 31 14:11:42.513199 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.513166    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4292c-openshift-default-c85866885-pk4x4"]
Oct 31 14:11:42.754153 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.754074    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dc7e2cf-7b12-4746-a474-705c25268e6a" path="/var/lib/kubelet/pods/0dc7e2cf-7b12-4746-a474-705c25268e6a/volumes"
Oct 31 14:11:42.754550 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.754537    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61628be-a3de-4cb7-aba5-d31e11396ad5" path="/var/lib/kubelet/pods/b61628be-a3de-4cb7-aba5-d31e11396ad5/volumes"
Oct 31 14:11:42.987364 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.987323    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv" podUID="963d02d0-8a8a-493f-8b2c-7b98b17f3413" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:11:42.987568 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:42.987409    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf" podUID="5a9df355-090e-4da6-a708-b7acbb252297" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:11:48.746026 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:11:48.745984    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:11:52.805383 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:52.805355    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"
Oct 31 14:11:52.833743 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:52.833707    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963d02d0-8a8a-493f-8b2c-7b98b17f3413-kube-api-access-6x9gd\" (UniqueName: \"kubernetes.io/projected/963d02d0-8a8a-493f-8b2c-7b98b17f3413-kube-api-access-6x9gd\") pod \"963d02d0-8a8a-493f-8b2c-7b98b17f3413\" (UID: \"963d02d0-8a8a-493f-8b2c-7b98b17f3413\") "
Oct 31 14:11:52.833961 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:52.833777    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963d02d0-8a8a-493f-8b2c-7b98b17f3413-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/963d02d0-8a8a-493f-8b2c-7b98b17f3413-plugins-config-volume\") pod \"963d02d0-8a8a-493f-8b2c-7b98b17f3413\" (UID: \"963d02d0-8a8a-493f-8b2c-7b98b17f3413\") "
Oct 31 14:11:52.834427 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:52.834270    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963d02d0-8a8a-493f-8b2c-7b98b17f3413-plugins-config-volume" pod "963d02d0-8a8a-493f-8b2c-7b98b17f3413" (UID: "963d02d0-8a8a-493f-8b2c-7b98b17f3413"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:11:52.837074 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:52.837045    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963d02d0-8a8a-493f-8b2c-7b98b17f3413-kube-api-access-6x9gd" pod "963d02d0-8a8a-493f-8b2c-7b98b17f3413" (UID: "963d02d0-8a8a-493f-8b2c-7b98b17f3413"). InnerVolumeSpecName "kube-api-access-6x9gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:11:52.839465 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:52.839448    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"
Oct 31 14:11:52.935452 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:52.935409    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9df355-090e-4da6-a708-b7acbb252297-kube-api-access-vxzlt\" (UniqueName: \"kubernetes.io/projected/5a9df355-090e-4da6-a708-b7acbb252297-kube-api-access-vxzlt\") pod \"5a9df355-090e-4da6-a708-b7acbb252297\" (UID: \"5a9df355-090e-4da6-a708-b7acbb252297\") "
Oct 31 14:11:52.935649 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:52.935500    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a9df355-090e-4da6-a708-b7acbb252297-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a9df355-090e-4da6-a708-b7acbb252297-plugins-config-volume\") pod \"5a9df355-090e-4da6-a708-b7acbb252297\" (UID: \"5a9df355-090e-4da6-a708-b7acbb252297\") "
Oct 31 14:11:52.935890 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:52.935851    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9df355-090e-4da6-a708-b7acbb252297-plugins-config-volume" pod "5a9df355-090e-4da6-a708-b7acbb252297" (UID: "5a9df355-090e-4da6-a708-b7acbb252297"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:11:52.935956 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:52.935937    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x9gd\" (UniqueName: \"kubernetes.io/projected/963d02d0-8a8a-493f-8b2c-7b98b17f3413-kube-api-access-6x9gd\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:52.935997 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:52.935961    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/963d02d0-8a8a-493f-8b2c-7b98b17f3413-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:52.937899 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:52.937862    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9df355-090e-4da6-a708-b7acbb252297-kube-api-access-vxzlt" pod "5a9df355-090e-4da6-a708-b7acbb252297" (UID: "5a9df355-090e-4da6-a708-b7acbb252297"). InnerVolumeSpecName "kube-api-access-vxzlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:11:53.037022 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.036934    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxzlt\" (UniqueName: \"kubernetes.io/projected/5a9df355-090e-4da6-a708-b7acbb252297-kube-api-access-vxzlt\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:53.037022 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.036967    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a9df355-090e-4da6-a708-b7acbb252297-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:11:53.545240 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.545210    4428 generic.go:356] "Generic (PLEG): container finished" podID="5a9df355-090e-4da6-a708-b7acbb252297" containerID="ad81c708dd8ea546ffc8d7a1362f86b6cc4cc53db18775e9e62028961313b052" exitCode=0
Oct 31 14:11:53.545387 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.545260    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"
Oct 31 14:11:53.545387 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.545289    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf" event={"ID":"5a9df355-090e-4da6-a708-b7acbb252297","Type":"ContainerDied","Data":"ad81c708dd8ea546ffc8d7a1362f86b6cc4cc53db18775e9e62028961313b052"}
Oct 31 14:11:53.545387 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.545344    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf" event={"ID":"5a9df355-090e-4da6-a708-b7acbb252297","Type":"ContainerDied","Data":"b97f8cda306992bb52f80fd2e5d6a64bb2e52ff1fa8e514294b3444f08b0dfdd"}
Oct 31 14:11:53.545387 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.545373    4428 scope.go:122] "RemoveContainer" containerID="ad81c708dd8ea546ffc8d7a1362f86b6cc4cc53db18775e9e62028961313b052"
Oct 31 14:11:53.546918 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.546856    4428 generic.go:356] "Generic (PLEG): container finished" podID="963d02d0-8a8a-493f-8b2c-7b98b17f3413" containerID="d1cc9b6197c08705ad5599147e7d47ef37fdd3a340d364c702aa2ea55d563457" exitCode=0
Oct 31 14:11:53.547058 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.546921    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"
Oct 31 14:11:53.547058 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.546976    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv" event={"ID":"963d02d0-8a8a-493f-8b2c-7b98b17f3413","Type":"ContainerDied","Data":"d1cc9b6197c08705ad5599147e7d47ef37fdd3a340d364c702aa2ea55d563457"}
Oct 31 14:11:53.547058 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.547020    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv" event={"ID":"963d02d0-8a8a-493f-8b2c-7b98b17f3413","Type":"ContainerDied","Data":"7a47690a3edb4a980a24877f154ef8b2456562794818342ef7ab2b1d83c6005b"}
Oct 31 14:11:53.555248 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.555228    4428 scope.go:122] "RemoveContainer" containerID="ad81c708dd8ea546ffc8d7a1362f86b6cc4cc53db18775e9e62028961313b052"
Oct 31 14:11:53.555595 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:11:53.555571    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad81c708dd8ea546ffc8d7a1362f86b6cc4cc53db18775e9e62028961313b052\": container with ID starting with ad81c708dd8ea546ffc8d7a1362f86b6cc4cc53db18775e9e62028961313b052 not found: ID does not exist" containerID="ad81c708dd8ea546ffc8d7a1362f86b6cc4cc53db18775e9e62028961313b052"
Oct 31 14:11:53.555711 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.555610    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad81c708dd8ea546ffc8d7a1362f86b6cc4cc53db18775e9e62028961313b052"} err="failed to get container status \"ad81c708dd8ea546ffc8d7a1362f86b6cc4cc53db18775e9e62028961313b052\": rpc error: code = NotFound desc = could not find container \"ad81c708dd8ea546ffc8d7a1362f86b6cc4cc53db18775e9e62028961313b052\": container with ID starting with ad81c708dd8ea546ffc8d7a1362f86b6cc4cc53db18775e9e62028961313b052 not found: ID does not exist"
Oct 31 14:11:53.555711 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.555632    4428 scope.go:122] "RemoveContainer" containerID="d1cc9b6197c08705ad5599147e7d47ef37fdd3a340d364c702aa2ea55d563457"
Oct 31 14:11:53.565143 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.565121    4428 scope.go:122] "RemoveContainer" containerID="d1cc9b6197c08705ad5599147e7d47ef37fdd3a340d364c702aa2ea55d563457"
Oct 31 14:11:53.565462 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:11:53.565442    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1cc9b6197c08705ad5599147e7d47ef37fdd3a340d364c702aa2ea55d563457\": container with ID starting with d1cc9b6197c08705ad5599147e7d47ef37fdd3a340d364c702aa2ea55d563457 not found: ID does not exist" containerID="d1cc9b6197c08705ad5599147e7d47ef37fdd3a340d364c702aa2ea55d563457"
Oct 31 14:11:53.565528 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.565471    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1cc9b6197c08705ad5599147e7d47ef37fdd3a340d364c702aa2ea55d563457"} err="failed to get container status \"d1cc9b6197c08705ad5599147e7d47ef37fdd3a340d364c702aa2ea55d563457\": rpc error: code = NotFound desc = could not find container \"d1cc9b6197c08705ad5599147e7d47ef37fdd3a340d364c702aa2ea55d563457\": container with ID starting with d1cc9b6197c08705ad5599147e7d47ef37fdd3a340d364c702aa2ea55d563457 not found: ID does not exist"
Oct 31 14:11:53.571594 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.571565    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"]
Oct 31 14:11:53.579833 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.579794    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c4292c-5486c2qzvf"]
Oct 31 14:11:53.599056 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.599027    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"]
Oct 31 14:11:53.604336 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:53.604310    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c4292c-c45b8d99cmjnzv"]
Oct 31 14:11:54.754148 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:54.754105    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9df355-090e-4da6-a708-b7acbb252297" path="/var/lib/kubelet/pods/5a9df355-090e-4da6-a708-b7acbb252297/volumes"
Oct 31 14:11:54.754698 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:54.754656    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963d02d0-8a8a-493f-8b2c-7b98b17f3413" path="/var/lib/kubelet/pods/963d02d0-8a8a-493f-8b2c-7b98b17f3413/volumes"
Oct 31 14:11:58.602458 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:11:58.602414    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:12:00.557431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:00.557394    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:12:06.733638 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:06.733604    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:12:08.045064 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:08.045027    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"]
Oct 31 14:12:08.048919 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:08.048887    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-42zxk"]
Oct 31 14:12:08.754215 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:08.754180    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43f7c0a1-2dee-4b55-8dcd-1cba4a981839" path="/var/lib/kubelet/pods/43f7c0a1-2dee-4b55-8dcd-1cba4a981839/volumes"
Oct 31 14:12:16.319050 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:16.318994    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:12:18.040227 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:18.040185    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"]
Oct 31 14:12:18.046035 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:18.046010    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8fjkk"]
Oct 31 14:12:18.741090 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:12:18.741054    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:12:18.754341 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:18.754308    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff36a20-8c96-4014-84a5-0f5be5a87e53" path="/var/lib/kubelet/pods/bff36a20-8c96-4014-84a5-0f5be5a87e53/volumes"
Oct 31 14:12:21.687625 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:21.687593    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:12:36.253854 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:36.253809    4428 scope.go:122] "RemoveContainer" containerID="d314738e96c05b1452593c62cf4077106e72bc73c315b1eb2f77e3320f0e06ac"
Oct 31 14:12:36.264376 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:36.264347    4428 scope.go:122] "RemoveContainer" containerID="91ca0903c37e84160f1d80bd9e9f7b6e59933c85abcd5a2a4211408763d1fe04"
Oct 31 14:12:39.028131 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:39.028079    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"]
Oct 31 14:12:39.031724 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:39.031697    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4xnz4"]
Oct 31 14:12:40.754075 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:40.754043    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff96a69f-cdae-4bb0-9575-a5a685a300e6" path="/var/lib/kubelet/pods/ff96a69f-cdae-4bb0-9575-a5a685a300e6/volumes"
Oct 31 14:12:48.741220 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:12:48.741179    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:12:58.029662 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:58.029621    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"]
Oct 31 14:12:58.033423 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:58.033390    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n7nf6"]
Oct 31 14:12:58.753980 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:12:58.753945    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5348bb10-c028-49c7-9c11-6f90f6f1e63d" path="/var/lib/kubelet/pods/5348bb10-c028-49c7-9c11-6f90f6f1e63d/volumes"
Oct 31 14:13:00.668466 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.668428    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cpbtr"]
Oct 31 14:13:00.668922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.668836    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b61628be-a3de-4cb7-aba5-d31e11396ad5" containerName="istio-proxy"
Oct 31 14:13:00.668922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.668846    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61628be-a3de-4cb7-aba5-d31e11396ad5" containerName="istio-proxy"
Oct 31 14:13:00.668922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.668856    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b61628be-a3de-4cb7-aba5-d31e11396ad5" containerName="istio-proxy"
Oct 31 14:13:00.668922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.668863    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61628be-a3de-4cb7-aba5-d31e11396ad5" containerName="istio-proxy"
Oct 31 14:13:00.668922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.668884    4428 cpu_manager.go:414] "containerMap: removing container" podUID="5a9df355-090e-4da6-a708-b7acbb252297" containerName="payload-pre-processing"
Oct 31 14:13:00.668922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.668890    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9df355-090e-4da6-a708-b7acbb252297" containerName="payload-pre-processing"
Oct 31 14:13:00.668922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.668901    4428 cpu_manager.go:414] "containerMap: removing container" podUID="963d02d0-8a8a-493f-8b2c-7b98b17f3413" containerName="payload-processing"
Oct 31 14:13:00.668922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.668906    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963d02d0-8a8a-493f-8b2c-7b98b17f3413" containerName="payload-processing"
Oct 31 14:13:00.668922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.668916    4428 cpu_manager.go:414] "containerMap: removing container" podUID="a239bc40-aa03-40ae-b7ff-c33ef400f867" containerName="revoke-keys"
Oct 31 14:13:00.668922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.668920    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a239bc40-aa03-40ae-b7ff-c33ef400f867" containerName="revoke-keys"
Oct 31 14:13:00.669285 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.668987    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61628be-a3de-4cb7-aba5-d31e11396ad5" containerName="istio-proxy"
Oct 31 14:13:00.669285 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.668994    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61628be-a3de-4cb7-aba5-d31e11396ad5" containerName="istio-proxy"
Oct 31 14:13:00.669285 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.669001    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9df355-090e-4da6-a708-b7acbb252297" containerName="payload-pre-processing"
Oct 31 14:13:00.669285 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.669008    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="963d02d0-8a8a-493f-8b2c-7b98b17f3413" containerName="payload-processing"
Oct 31 14:13:00.669285 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.669014    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a239bc40-aa03-40ae-b7ff-c33ef400f867" containerName="revoke-keys"
Oct 31 14:13:00.673771 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.673751    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cpbtr"
Oct 31 14:13:00.681015 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.680993    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 14:13:00.681524 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.681496    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 14:13:00.681834 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.681820    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ztmv6\""
Oct 31 14:13:00.688955 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.688930    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cpbtr"]
Oct 31 14:13:00.781911 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.781848    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsc59\" (UniqueName: \"kubernetes.io/projected/4ada56cc-ea77-4a96-bb9b-3e26162a1aa6-kube-api-access-dsc59\") pod \"kuadrant-operator-controller-manager-678d87766f-cpbtr\" (UID: \"4ada56cc-ea77-4a96-bb9b-3e26162a1aa6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cpbtr"
Oct 31 14:13:00.782121 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.782032    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ada56cc-ea77-4a96-bb9b-3e26162a1aa6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cpbtr\" (UID: \"4ada56cc-ea77-4a96-bb9b-3e26162a1aa6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cpbtr"
Oct 31 14:13:00.883138 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.883091    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsc59\" (UniqueName: \"kubernetes.io/projected/4ada56cc-ea77-4a96-bb9b-3e26162a1aa6-kube-api-access-dsc59\") pod \"kuadrant-operator-controller-manager-678d87766f-cpbtr\" (UID: \"4ada56cc-ea77-4a96-bb9b-3e26162a1aa6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cpbtr"
Oct 31 14:13:00.883344 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.883257    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ada56cc-ea77-4a96-bb9b-3e26162a1aa6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cpbtr\" (UID: \"4ada56cc-ea77-4a96-bb9b-3e26162a1aa6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cpbtr"
Oct 31 14:13:00.883659 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.883639    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ada56cc-ea77-4a96-bb9b-3e26162a1aa6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cpbtr\" (UID: \"4ada56cc-ea77-4a96-bb9b-3e26162a1aa6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cpbtr"
Oct 31 14:13:00.897333 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.897302    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsc59\" (UniqueName: \"kubernetes.io/projected/4ada56cc-ea77-4a96-bb9b-3e26162a1aa6-kube-api-access-dsc59\") pod \"kuadrant-operator-controller-manager-678d87766f-cpbtr\" (UID: \"4ada56cc-ea77-4a96-bb9b-3e26162a1aa6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cpbtr"
Oct 31 14:13:00.987301 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:00.987264    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cpbtr"
Oct 31 14:13:01.278002 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:01.277976    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cpbtr"]
Oct 31 14:13:01.280806 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:13:01.280777    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ada56cc_ea77_4a96_bb9b_3e26162a1aa6.slice/crio-560495fb4ff0c98c5a83ce7a9f8314631861bc4c3c1169c0b189f3931bfcb74e WatchSource:0}: Error finding container 560495fb4ff0c98c5a83ce7a9f8314631861bc4c3c1169c0b189f3931bfcb74e: Status 404 returned error can't find the container with id 560495fb4ff0c98c5a83ce7a9f8314631861bc4c3c1169c0b189f3931bfcb74e
Oct 31 14:13:01.920593 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:01.920551    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cpbtr" event={"ID":"4ada56cc-ea77-4a96-bb9b-3e26162a1aa6","Type":"ContainerStarted","Data":"560495fb4ff0c98c5a83ce7a9f8314631861bc4c3c1169c0b189f3931bfcb74e"}
Oct 31 14:13:05.945904 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:05.945851    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cpbtr" event={"ID":"4ada56cc-ea77-4a96-bb9b-3e26162a1aa6","Type":"ContainerStarted","Data":"21e249ac73ff7dc1f5ad5ee72b8d8fc31289a894b06422e0c8b1439fbc1f00ca"}
Oct 31 14:13:05.946334 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:05.945993    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cpbtr"
Oct 31 14:13:05.972372 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:05.972311    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cpbtr" podStartSLOduration=1.818771889 podStartE2EDuration="5.97229377s" totalImagesPullingTime="4.153521881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:13:05.968987033 +0000 UTC m=+4183.367335658" watchObservedRunningTime="2026-10-31 14:13:05.97229377 +0000 UTC m=+4183.370642643"
Oct 31 14:13:16.397644 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:16.397603    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:13:16.953591 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:16.953558    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cpbtr"
Oct 31 14:13:17.052312 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:17.052275    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"]
Oct 31 14:13:17.055577 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:17.055546    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9zkb7"]
Oct 31 14:13:18.671184 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:18.671148    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:13:18.740994 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:13:18.740958    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:13:18.754095 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:18.754068    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5899d096-c8b6-4b3c-979c-45e981e5acc8" path="/var/lib/kubelet/pods/5899d096-c8b6-4b3c-979c-45e981e5acc8/volumes"
Oct 31 14:13:18.837741 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:18.837707    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:13:28.022875 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:28.022838    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:13:28.905550 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:28.905511    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"]
Oct 31 14:13:28.910324 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:28.910304    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"
Oct 31 14:13:28.919926 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:28.919897    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:13:28.920105 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:28.919951    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:13:28.922318 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:28.922294    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"]
Oct 31 14:13:28.976883 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:28.976842    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa58b67-cc9e-4608-a17c-330435693cb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-nn2gt\" (UID: \"aaa58b67-cc9e-4608-a17c-330435693cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"
Oct 31 14:13:28.977076 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:28.976965    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa58b67-cc9e-4608-a17c-330435693cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-nn2gt\" (UID: \"aaa58b67-cc9e-4608-a17c-330435693cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"
Oct 31 14:13:28.977076 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:28.977020    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp9m6\" (UniqueName: \"kubernetes.io/projected/aaa58b67-cc9e-4608-a17c-330435693cb5-kube-api-access-lp9m6\") pod \"maas-api-revoke-sub-d576e68b-nn2gt\" (UID: \"aaa58b67-cc9e-4608-a17c-330435693cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"
Oct 31 14:13:29.078476 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:29.078436    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa58b67-cc9e-4608-a17c-330435693cb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-nn2gt\" (UID: \"aaa58b67-cc9e-4608-a17c-330435693cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"
Oct 31 14:13:29.078934 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:29.078495    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa58b67-cc9e-4608-a17c-330435693cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-nn2gt\" (UID: \"aaa58b67-cc9e-4608-a17c-330435693cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"
Oct 31 14:13:29.078934 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:29.078530    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp9m6\" (UniqueName: \"kubernetes.io/projected/aaa58b67-cc9e-4608-a17c-330435693cb5-kube-api-access-lp9m6\") pod \"maas-api-revoke-sub-d576e68b-nn2gt\" (UID: \"aaa58b67-cc9e-4608-a17c-330435693cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"
Oct 31 14:13:29.079256 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:29.079224    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa58b67-cc9e-4608-a17c-330435693cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-nn2gt\" (UID: \"aaa58b67-cc9e-4608-a17c-330435693cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"
Oct 31 14:13:29.081035 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:29.081010    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa58b67-cc9e-4608-a17c-330435693cb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-nn2gt\" (UID: \"aaa58b67-cc9e-4608-a17c-330435693cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"
Oct 31 14:13:29.094047 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:29.094012    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp9m6\" (UniqueName: \"kubernetes.io/projected/aaa58b67-cc9e-4608-a17c-330435693cb5-kube-api-access-lp9m6\") pod \"maas-api-revoke-sub-d576e68b-nn2gt\" (UID: \"aaa58b67-cc9e-4608-a17c-330435693cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"
Oct 31 14:13:29.224459 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:29.224424    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"
Oct 31 14:13:29.505621 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:29.505525    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"]
Oct 31 14:13:29.509370 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:13:29.509334    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa58b67_cc9e_4608_a17c_330435693cb5.slice/crio-0783bb516efa8f35e0cc67d58159082a293d6035ef47af065dc9abe67557e128 WatchSource:0}: Error finding container 0783bb516efa8f35e0cc67d58159082a293d6035ef47af065dc9abe67557e128: Status 404 returned error can't find the container with id 0783bb516efa8f35e0cc67d58159082a293d6035ef47af065dc9abe67557e128
Oct 31 14:13:30.095277 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:30.095236    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt" event={"ID":"aaa58b67-cc9e-4608-a17c-330435693cb5","Type":"ContainerStarted","Data":"a138dde338fbd1e66d5dd1c0729d60b0e664641920244f193f1e15de9f67ec90"}
Oct 31 14:13:30.095784 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:30.095292    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt" event={"ID":"aaa58b67-cc9e-4608-a17c-330435693cb5","Type":"ContainerStarted","Data":"0783bb516efa8f35e0cc67d58159082a293d6035ef47af065dc9abe67557e128"}
Oct 31 14:13:30.118152 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:30.118008    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt" podStartSLOduration=2.117971197 podStartE2EDuration="2.117971197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:13:30.115441267 +0000 UTC m=+4207.513789891" watchObservedRunningTime="2026-10-31 14:13:30.117971197 +0000 UTC m=+4207.516319821"
Oct 31 14:13:31.102043 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:31.102003    4428 generic.go:356] "Generic (PLEG): container finished" podID="aaa58b67-cc9e-4608-a17c-330435693cb5" containerID="a138dde338fbd1e66d5dd1c0729d60b0e664641920244f193f1e15de9f67ec90" exitCode=0
Oct 31 14:13:31.102478 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:31.102091    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt" event={"ID":"aaa58b67-cc9e-4608-a17c-330435693cb5","Type":"ContainerDied","Data":"a138dde338fbd1e66d5dd1c0729d60b0e664641920244f193f1e15de9f67ec90"}
Oct 31 14:13:32.252004 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:32.251974    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"
Oct 31 14:13:32.311691 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:32.311612    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa58b67-cc9e-4608-a17c-330435693cb5-kube-api-access-lp9m6\" (UniqueName: \"kubernetes.io/projected/aaa58b67-cc9e-4608-a17c-330435693cb5-kube-api-access-lp9m6\") pod \"aaa58b67-cc9e-4608-a17c-330435693cb5\" (UID: \"aaa58b67-cc9e-4608-a17c-330435693cb5\") "
Oct 31 14:13:32.312125 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:32.311715    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaa58b67-cc9e-4608-a17c-330435693cb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa58b67-cc9e-4608-a17c-330435693cb5-maas-api-xks-ca\") pod \"aaa58b67-cc9e-4608-a17c-330435693cb5\" (UID: \"aaa58b67-cc9e-4608-a17c-330435693cb5\") "
Oct 31 14:13:32.312125 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:32.311933    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaa58b67-cc9e-4608-a17c-330435693cb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa58b67-cc9e-4608-a17c-330435693cb5-maas-api-openshift-ca\") pod \"aaa58b67-cc9e-4608-a17c-330435693cb5\" (UID: \"aaa58b67-cc9e-4608-a17c-330435693cb5\") "
Oct 31 14:13:32.312393 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:32.312358    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa58b67-cc9e-4608-a17c-330435693cb5-maas-api-openshift-ca" pod "aaa58b67-cc9e-4608-a17c-330435693cb5" (UID: "aaa58b67-cc9e-4608-a17c-330435693cb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:13:32.312494 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:32.312475    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa58b67-cc9e-4608-a17c-330435693cb5-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:13:32.315739 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:32.315695    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa58b67-cc9e-4608-a17c-330435693cb5-maas-api-xks-ca" pod "aaa58b67-cc9e-4608-a17c-330435693cb5" (UID: "aaa58b67-cc9e-4608-a17c-330435693cb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:13:32.315739 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:32.315717    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa58b67-cc9e-4608-a17c-330435693cb5-kube-api-access-lp9m6" pod "aaa58b67-cc9e-4608-a17c-330435693cb5" (UID: "aaa58b67-cc9e-4608-a17c-330435693cb5"). InnerVolumeSpecName "kube-api-access-lp9m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:13:32.414025 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:32.413932    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp9m6\" (UniqueName: \"kubernetes.io/projected/aaa58b67-cc9e-4608-a17c-330435693cb5-kube-api-access-lp9m6\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:13:32.414025 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:32.413961    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa58b67-cc9e-4608-a17c-330435693cb5-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:13:33.121280 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:33.121188    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"
Oct 31 14:13:33.121280 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:33.121198    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt" event={"ID":"aaa58b67-cc9e-4608-a17c-330435693cb5","Type":"ContainerDied","Data":"0783bb516efa8f35e0cc67d58159082a293d6035ef47af065dc9abe67557e128"}
Oct 31 14:13:33.121280 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:33.121274    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0783bb516efa8f35e0cc67d58159082a293d6035ef47af065dc9abe67557e128"
Oct 31 14:13:34.125956 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:34.125854    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"]
Oct 31 14:13:34.135829 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:34.135792    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7mn22"]
Oct 31 14:13:34.754618 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:34.754587    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91575831-49a4-41c4-842b-34c7f6fceb3c" path="/var/lib/kubelet/pods/91575831-49a4-41c4-842b-34c7f6fceb3c/volumes"
Oct 31 14:13:36.311348 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:36.311311    4428 scope.go:122] "RemoveContainer" containerID="0bcd258fcf2804a89d13ff5ca15ee48f54469aec65a73971cde45d212c32564f"
Oct 31 14:13:36.321125 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:36.321100    4428 scope.go:122] "RemoveContainer" containerID="c7c3afc7115c09af269e3be6b4fa2d7502c2c6f1b121b8d49ad03fc08560a4b0"
Oct 31 14:13:36.329754 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:36.329727    4428 scope.go:122] "RemoveContainer" containerID="18e0ad522119af8861bc16a5241525233ea7210cd69b3280100a2f746c4295bb"
Oct 31 14:13:36.339101 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:36.339078    4428 scope.go:122] "RemoveContainer" containerID="6fd7483e8ecb12f8b5999b8a1f485c2ce98826fa5cd5199c28d4898c29a4e556"
Oct 31 14:13:48.741588 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:13:48.741545    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:13:50.077301 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.077267    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"]
Oct 31 14:13:50.077715 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.077673    4428 cpu_manager.go:414] "containerMap: removing container" podUID="aaa58b67-cc9e-4608-a17c-330435693cb5" containerName="revoke-keys"
Oct 31 14:13:50.077715 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.077684    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa58b67-cc9e-4608-a17c-330435693cb5" containerName="revoke-keys"
Oct 31 14:13:50.077797 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.077758    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa58b67-cc9e-4608-a17c-330435693cb5" containerName="revoke-keys"
Oct 31 14:13:50.082970 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.082950    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"
Oct 31 14:13:50.088183 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.088152    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:13:50.088183 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.088152    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:13:50.090010 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.089986    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"]
Oct 31 14:13:50.197963 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.197923    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45f885f-cf08-4803-b7e1-6a16d279ff8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-b45f9\" (UID: \"c45f885f-cf08-4803-b7e1-6a16d279ff8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"
Oct 31 14:13:50.198164 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.197980    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45f885f-cf08-4803-b7e1-6a16d279ff8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-b45f9\" (UID: \"c45f885f-cf08-4803-b7e1-6a16d279ff8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"
Oct 31 14:13:50.198164 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.198123    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9ptd\" (UniqueName: \"kubernetes.io/projected/c45f885f-cf08-4803-b7e1-6a16d279ff8a-kube-api-access-d9ptd\") pod \"maas-api-revoke-sub-d2bc93bc-b45f9\" (UID: \"c45f885f-cf08-4803-b7e1-6a16d279ff8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"
Oct 31 14:13:50.299112 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.299070    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45f885f-cf08-4803-b7e1-6a16d279ff8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-b45f9\" (UID: \"c45f885f-cf08-4803-b7e1-6a16d279ff8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"
Oct 31 14:13:50.299112 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.299120    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45f885f-cf08-4803-b7e1-6a16d279ff8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-b45f9\" (UID: \"c45f885f-cf08-4803-b7e1-6a16d279ff8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"
Oct 31 14:13:50.299350 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.299171    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9ptd\" (UniqueName: \"kubernetes.io/projected/c45f885f-cf08-4803-b7e1-6a16d279ff8a-kube-api-access-d9ptd\") pod \"maas-api-revoke-sub-d2bc93bc-b45f9\" (UID: \"c45f885f-cf08-4803-b7e1-6a16d279ff8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"
Oct 31 14:13:50.299838 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.299814    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45f885f-cf08-4803-b7e1-6a16d279ff8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-b45f9\" (UID: \"c45f885f-cf08-4803-b7e1-6a16d279ff8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"
Oct 31 14:13:50.301630 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.301610    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45f885f-cf08-4803-b7e1-6a16d279ff8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-b45f9\" (UID: \"c45f885f-cf08-4803-b7e1-6a16d279ff8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"
Oct 31 14:13:50.313045 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.313018    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9ptd\" (UniqueName: \"kubernetes.io/projected/c45f885f-cf08-4803-b7e1-6a16d279ff8a-kube-api-access-d9ptd\") pod \"maas-api-revoke-sub-d2bc93bc-b45f9\" (UID: \"c45f885f-cf08-4803-b7e1-6a16d279ff8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"
Oct 31 14:13:50.398355 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.398256    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"
Oct 31 14:13:50.676968 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:50.676940    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"]
Oct 31 14:13:50.678594 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:13:50.678555    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45f885f_cf08_4803_b7e1_6a16d279ff8a.slice/crio-58567ca990bf078a2fabad98df0b32d2d97894e290e752f200df37111c29a3a1 WatchSource:0}: Error finding container 58567ca990bf078a2fabad98df0b32d2d97894e290e752f200df37111c29a3a1: Status 404 returned error can't find the container with id 58567ca990bf078a2fabad98df0b32d2d97894e290e752f200df37111c29a3a1
Oct 31 14:13:51.243580 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:51.243535    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9" event={"ID":"c45f885f-cf08-4803-b7e1-6a16d279ff8a","Type":"ContainerStarted","Data":"b05495d789219ab0131dba9bf011c93707c7aadc46016111d33cf74418b3efd0"}
Oct 31 14:13:51.244048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:51.243603    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9" event={"ID":"c45f885f-cf08-4803-b7e1-6a16d279ff8a","Type":"ContainerStarted","Data":"58567ca990bf078a2fabad98df0b32d2d97894e290e752f200df37111c29a3a1"}
Oct 31 14:13:51.265891 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:51.265774    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9" podStartSLOduration=1.265750352 podStartE2EDuration="1.265750352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:13:51.264566779 +0000 UTC m=+4228.662915404" watchObservedRunningTime="2026-10-31 14:13:51.265750352 +0000 UTC m=+4228.664098976"
Oct 31 14:13:52.249644 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:52.249603    4428 generic.go:356] "Generic (PLEG): container finished" podID="c45f885f-cf08-4803-b7e1-6a16d279ff8a" containerID="b05495d789219ab0131dba9bf011c93707c7aadc46016111d33cf74418b3efd0" exitCode=0
Oct 31 14:13:52.250119 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:52.249688    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9" event={"ID":"c45f885f-cf08-4803-b7e1-6a16d279ff8a","Type":"ContainerDied","Data":"b05495d789219ab0131dba9bf011c93707c7aadc46016111d33cf74418b3efd0"}
Oct 31 14:13:53.396990 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:53.396966    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"
Oct 31 14:13:53.530963 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:53.530854    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c45f885f-cf08-4803-b7e1-6a16d279ff8a-kube-api-access-d9ptd\" (UniqueName: \"kubernetes.io/projected/c45f885f-cf08-4803-b7e1-6a16d279ff8a-kube-api-access-d9ptd\") pod \"c45f885f-cf08-4803-b7e1-6a16d279ff8a\" (UID: \"c45f885f-cf08-4803-b7e1-6a16d279ff8a\") "
Oct 31 14:13:53.531145 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:53.530973    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c45f885f-cf08-4803-b7e1-6a16d279ff8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45f885f-cf08-4803-b7e1-6a16d279ff8a-maas-api-openshift-ca\") pod \"c45f885f-cf08-4803-b7e1-6a16d279ff8a\" (UID: \"c45f885f-cf08-4803-b7e1-6a16d279ff8a\") "
Oct 31 14:13:53.531145 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:53.531035    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c45f885f-cf08-4803-b7e1-6a16d279ff8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45f885f-cf08-4803-b7e1-6a16d279ff8a-maas-api-xks-ca\") pod \"c45f885f-cf08-4803-b7e1-6a16d279ff8a\" (UID: \"c45f885f-cf08-4803-b7e1-6a16d279ff8a\") "
Oct 31 14:13:53.531405 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:53.531375    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45f885f-cf08-4803-b7e1-6a16d279ff8a-maas-api-openshift-ca" pod "c45f885f-cf08-4803-b7e1-6a16d279ff8a" (UID: "c45f885f-cf08-4803-b7e1-6a16d279ff8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:13:53.533362 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:53.533324    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c45f885f-cf08-4803-b7e1-6a16d279ff8a-maas-api-xks-ca" pod "c45f885f-cf08-4803-b7e1-6a16d279ff8a" (UID: "c45f885f-cf08-4803-b7e1-6a16d279ff8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:13:53.533501 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:53.533393    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45f885f-cf08-4803-b7e1-6a16d279ff8a-kube-api-access-d9ptd" pod "c45f885f-cf08-4803-b7e1-6a16d279ff8a" (UID: "c45f885f-cf08-4803-b7e1-6a16d279ff8a"). InnerVolumeSpecName "kube-api-access-d9ptd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:13:53.632446 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:53.632395    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45f885f-cf08-4803-b7e1-6a16d279ff8a-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:13:53.632446 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:53.632441    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45f885f-cf08-4803-b7e1-6a16d279ff8a-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:13:53.632446 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:53.632452    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9ptd\" (UniqueName: \"kubernetes.io/projected/c45f885f-cf08-4803-b7e1-6a16d279ff8a-kube-api-access-d9ptd\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:13:54.261258 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:54.261229    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"
Oct 31 14:13:54.261431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:54.261260    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9" event={"ID":"c45f885f-cf08-4803-b7e1-6a16d279ff8a","Type":"ContainerDied","Data":"58567ca990bf078a2fabad98df0b32d2d97894e290e752f200df37111c29a3a1"}
Oct 31 14:13:54.261431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:13:54.261312    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58567ca990bf078a2fabad98df0b32d2d97894e290e752f200df37111c29a3a1"
Oct 31 14:14:01.846620 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:01.846549    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:02.182432 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.182306    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"]
Oct 31 14:14:02.182842 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.182809    4428 cpu_manager.go:414] "containerMap: removing container" podUID="c45f885f-cf08-4803-b7e1-6a16d279ff8a" containerName="revoke-keys"
Oct 31 14:14:02.182842 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.182829    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c45f885f-cf08-4803-b7e1-6a16d279ff8a" containerName="revoke-keys"
Oct 31 14:14:02.183128 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.182922    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c45f885f-cf08-4803-b7e1-6a16d279ff8a" containerName="revoke-keys"
Oct 31 14:14:02.186283 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.186258    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.191801 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.191778    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-a9dd7f-openshift-default-dockercfg-zwfn5\""
Oct 31 14:14:02.200902 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.200839    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"]
Oct 31 14:14:02.284948 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.284909    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:02.319700 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.319660    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v9ns\" (UniqueName: \"kubernetes.io/projected/43a58059-e4ff-43d0-a312-2cb54fed07c3-kube-api-access-9v9ns\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.319958 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.319720    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-data\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.319958 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.319832    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-workload-socket\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.319958 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.319893    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-credential-socket\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.319958 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.319911    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-envoy\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.319958 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.319934    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-token\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.320276 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.319980    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a58059-e4ff-43d0-a312-2cb54fed07c3-istiod-ca-cert\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.320276 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.320007    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-podinfo\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.320276 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.320046    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-workload-certs\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.421592 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.421542    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-data\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.421818 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.421651    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-workload-socket\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.421818 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.421688    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-credential-socket\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.421818 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.421715    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-envoy\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.421818 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.421754    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-token\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.421818 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.421799    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a58059-e4ff-43d0-a312-2cb54fed07c3-istiod-ca-cert\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.422155 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.421825    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-podinfo\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.422155 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.421912    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-workload-certs\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.422155 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.422021    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v9ns\" (UniqueName: \"kubernetes.io/projected/43a58059-e4ff-43d0-a312-2cb54fed07c3-kube-api-access-9v9ns\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.422317 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.422189    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-data\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.422568 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.422520    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-workload-socket\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.422699 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.422582    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-credential-socket\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.424054 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.423035    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-workload-certs\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.424054 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.423497    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a58059-e4ff-43d0-a312-2cb54fed07c3-istiod-ca-cert\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.425807 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.425774    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-envoy\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.426165 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.426146    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-podinfo\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.436253 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.436192    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v9ns\" (UniqueName: \"kubernetes.io/projected/43a58059-e4ff-43d0-a312-2cb54fed07c3-kube-api-access-9v9ns\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.436365 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.436331    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-token\") pod \"e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") " pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.443623 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.443596    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:02.508555 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.508517    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:02.797317 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.797291    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"]
Oct 31 14:14:02.799518 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:14:02.799489    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a58059_e4ff_43d0_a312_2cb54fed07c3.slice/crio-fe10e17e0278374ad59ffc6136e8997f9810f5393e30fae90cb0009bb6f20deb WatchSource:0}: Error finding container fe10e17e0278374ad59ffc6136e8997f9810f5393e30fae90cb0009bb6f20deb: Status 404 returned error can't find the container with id fe10e17e0278374ad59ffc6136e8997f9810f5393e30fae90cb0009bb6f20deb
Oct 31 14:14:02.801779 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.801742    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:14:02.801961 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.801824    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:14:02.801961 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:02.801890    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:14:03.311171 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:03.311131    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5" event={"ID":"43a58059-e4ff-43d0-a312-2cb54fed07c3","Type":"ContainerStarted","Data":"1b506f95a081996f09a8c8dbfc21fc8852dc1de51f285dab4d610cc3a7231f1e"}
Oct 31 14:14:03.311646 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:03.311188    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5" event={"ID":"43a58059-e4ff-43d0-a312-2cb54fed07c3","Type":"ContainerStarted","Data":"fe10e17e0278374ad59ffc6136e8997f9810f5393e30fae90cb0009bb6f20deb"}
Oct 31 14:14:03.336879 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:03.336820    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5" podStartSLOduration=1.336803505 podStartE2EDuration="1.336803505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:14:03.336276238 +0000 UTC m=+4240.734624863" watchObservedRunningTime="2026-10-31 14:14:03.336803505 +0000 UTC m=+4240.735152125"
Oct 31 14:14:03.509666 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:03.509620    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:03.515046 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:03.515018    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:04.316560 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:04.316525    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:04.317712 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:04.317694    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:05.671556 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:05.671515    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:05.831747 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:05.831712    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:06.811576 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.811535    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"]
Oct 31 14:14:06.815778 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.815746    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:14:06.831952 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.831923    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"]
Oct 31 14:14:06.833968 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.833937    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"]
Oct 31 14:14:06.834150 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.834093    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-a9dd7f-serving-cert\""
Oct 31 14:14:06.834225 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.834159    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bx4t6\""
Oct 31 14:14:06.834281 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.834225    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 14:14:06.838964 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.838938    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"
Oct 31 14:14:06.846939 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.844762    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-a9dd7f\""
Oct 31 14:14:06.847967 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.847935    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"]
Oct 31 14:14:06.855208 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.855177    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"]
Oct 31 14:14:06.855467 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.855443    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"
Oct 31 14:14:06.857368 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.857344    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"]
Oct 31 14:14:06.867384 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.867356    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:06.872782 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.872747    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-709cb2ca-dockercfg-f27rl\""
Oct 31 14:14:06.968530 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.968497    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e37f1ca7-f48b-43be-9bd5-ba38e394653c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz\" (UID: \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:14:06.968718 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.968545    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kkd7\" (UniqueName: \"kubernetes.io/projected/21912d67-a608-4417-9191-df1dcc8d958c-kube-api-access-7kkd7\") pod \"payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf\" (UID: \"21912d67-a608-4417-9191-df1dcc8d958c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"
Oct 31 14:14:06.968718 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.968665    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e37f1ca7-f48b-43be-9bd5-ba38e394653c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz\" (UID: \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:14:06.968808 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.968725    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbsd2\" (UniqueName: \"kubernetes.io/projected/e37f1ca7-f48b-43be-9bd5-ba38e394653c-kube-api-access-cbsd2\") pod \"maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz\" (UID: \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:14:06.968808 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.968789    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e37f1ca7-f48b-43be-9bd5-ba38e394653c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz\" (UID: \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:14:06.968897 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.968815    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtdzq\" (UniqueName: \"kubernetes.io/projected/3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe-kube-api-access-rtdzq\") pod \"payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s\" (UID: \"3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"
Oct 31 14:14:06.968950 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.968929    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s\" (UID: \"3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"
Oct 31 14:14:06.968997 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:06.968979    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21912d67-a608-4417-9191-df1dcc8d958c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf\" (UID: \"21912d67-a608-4417-9191-df1dcc8d958c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"
Oct 31 14:14:07.035902 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.035852    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:07.070985 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.070726    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21912d67-a608-4417-9191-df1dcc8d958c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf\" (UID: \"21912d67-a608-4417-9191-df1dcc8d958c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"
Oct 31 14:14:07.070985 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.070813    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e37f1ca7-f48b-43be-9bd5-ba38e394653c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz\" (UID: \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:14:07.070985 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.070856    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kkd7\" (UniqueName: \"kubernetes.io/projected/21912d67-a608-4417-9191-df1dcc8d958c-kube-api-access-7kkd7\") pod \"payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf\" (UID: \"21912d67-a608-4417-9191-df1dcc8d958c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"
Oct 31 14:14:07.071496 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.070995    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e37f1ca7-f48b-43be-9bd5-ba38e394653c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz\" (UID: \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:14:07.071496 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.071093    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbsd2\" (UniqueName: \"kubernetes.io/projected/e37f1ca7-f48b-43be-9bd5-ba38e394653c-kube-api-access-cbsd2\") pod \"maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz\" (UID: \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:14:07.071496 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.071139    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e37f1ca7-f48b-43be-9bd5-ba38e394653c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz\" (UID: \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:14:07.071496 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.071170    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtdzq\" (UniqueName: \"kubernetes.io/projected/3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe-kube-api-access-rtdzq\") pod \"payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s\" (UID: \"3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"
Oct 31 14:14:07.071496 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.071261    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s\" (UID: \"3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"
Oct 31 14:14:07.072244 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.072199    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21912d67-a608-4417-9191-df1dcc8d958c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf\" (UID: \"21912d67-a608-4417-9191-df1dcc8d958c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"
Oct 31 14:14:07.072561 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.072524    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s\" (UID: \"3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"
Oct 31 14:14:07.072861 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.072817    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e37f1ca7-f48b-43be-9bd5-ba38e394653c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz\" (UID: \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:14:07.075512 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.075455    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e37f1ca7-f48b-43be-9bd5-ba38e394653c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz\" (UID: \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:14:07.075512 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.075481    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e37f1ca7-f48b-43be-9bd5-ba38e394653c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz\" (UID: \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:14:07.085063 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.085033    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kkd7\" (UniqueName: \"kubernetes.io/projected/21912d67-a608-4417-9191-df1dcc8d958c-kube-api-access-7kkd7\") pod \"payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf\" (UID: \"21912d67-a608-4417-9191-df1dcc8d958c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"
Oct 31 14:14:07.085357 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.085337    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtdzq\" (UniqueName: \"kubernetes.io/projected/3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe-kube-api-access-rtdzq\") pod \"payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s\" (UID: \"3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"
Oct 31 14:14:07.085518 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.085494    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbsd2\" (UniqueName: \"kubernetes.io/projected/e37f1ca7-f48b-43be-9bd5-ba38e394653c-kube-api-access-cbsd2\") pod \"maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz\" (UID: \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:14:07.137553 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.137516    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:14:07.159947 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.159912    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"
Oct 31 14:14:07.171172 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.171146    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"
Oct 31 14:14:07.203564 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.203419    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:07.424532 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.424503    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"]
Oct 31 14:14:07.426530 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:14:07.426502    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37f1ca7_f48b_43be_9bd5_ba38e394653c.slice/crio-3b5ea34878f88a9886c231e1f631edfb5664696a9dc7d9b89e1a4e25724f9e39 WatchSource:0}: Error finding container 3b5ea34878f88a9886c231e1f631edfb5664696a9dc7d9b89e1a4e25724f9e39: Status 404 returned error can't find the container with id 3b5ea34878f88a9886c231e1f631edfb5664696a9dc7d9b89e1a4e25724f9e39
Oct 31 14:14:07.478061 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.478027    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"]
Oct 31 14:14:07.479403 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:14:07.479377    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21912d67_a608_4417_9191_df1dcc8d958c.slice/crio-25e8c97407b1cccd63e9482b983f781c5a5e8163e338bc5f90e9ea2812d50f4f WatchSource:0}: Error finding container 25e8c97407b1cccd63e9482b983f781c5a5e8163e338bc5f90e9ea2812d50f4f: Status 404 returned error can't find the container with id 25e8c97407b1cccd63e9482b983f781c5a5e8163e338bc5f90e9ea2812d50f4f
Oct 31 14:14:07.491988 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.491962    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"]
Oct 31 14:14:07.493701 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:14:07.493678    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2dc027_89b9_4cf9_b96d_bfb87c4d01fe.slice/crio-472d1eebcfed1eb67304a9cc2777c87c603418f6f831274bb6e108a44931468f WatchSource:0}: Error finding container 472d1eebcfed1eb67304a9cc2777c87c603418f6f831274bb6e108a44931468f: Status 404 returned error can't find the container with id 472d1eebcfed1eb67304a9cc2777c87c603418f6f831274bb6e108a44931468f
Oct 31 14:14:07.755012 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:07.754977    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:08.351172 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:08.351057    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s" event={"ID":"3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe","Type":"ContainerStarted","Data":"1e9949005068167b37a79bde2324f9a7546addb54a398e45484b922a74e70472"}
Oct 31 14:14:08.351639 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:08.351259    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"
Oct 31 14:14:08.351639 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:08.351320    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s" event={"ID":"3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe","Type":"ContainerStarted","Data":"472d1eebcfed1eb67304a9cc2777c87c603418f6f831274bb6e108a44931468f"}
Oct 31 14:14:08.352918 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:08.352892    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz" event={"ID":"e37f1ca7-f48b-43be-9bd5-ba38e394653c","Type":"ContainerStarted","Data":"13a22cf4624aa531151e848decb762cb37d3a281c8f5e0c58148c3c54c0010c9"}
Oct 31 14:14:08.353067 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:08.352931    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz" event={"ID":"e37f1ca7-f48b-43be-9bd5-ba38e394653c","Type":"ContainerStarted","Data":"3b5ea34878f88a9886c231e1f631edfb5664696a9dc7d9b89e1a4e25724f9e39"}
Oct 31 14:14:08.353067 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:08.353026    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:14:08.354671 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:08.354637    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf" event={"ID":"21912d67-a608-4417-9191-df1dcc8d958c","Type":"ContainerStarted","Data":"e15c8732e20f1bfffcb329b1c104f1692c77be6852cab8fdc45d2b266f16228c"}
Oct 31 14:14:08.354789 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:08.354698    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf" event={"ID":"21912d67-a608-4417-9191-df1dcc8d958c","Type":"ContainerStarted","Data":"25e8c97407b1cccd63e9482b983f781c5a5e8163e338bc5f90e9ea2812d50f4f"}
Oct 31 14:14:08.354789 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:08.354732    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"
Oct 31 14:14:08.373410 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:08.373345    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s" podStartSLOduration=2.373325814 podStartE2EDuration="2.373325814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:14:08.369919179 +0000 UTC m=+4245.768267805" watchObservedRunningTime="2026-10-31 14:14:08.373325814 +0000 UTC m=+4245.771674439"
Oct 31 14:14:08.393788 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:08.393728    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz" podStartSLOduration=1.863737242 podStartE2EDuration="2.393714031s" totalImagesPullingTime="529.976789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:14:08.391331772 +0000 UTC m=+4245.789680396" watchObservedRunningTime="2026-10-31 14:14:08.393714031 +0000 UTC m=+4245.792062646"
Oct 31 14:14:08.418290 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:08.418235    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf" podStartSLOduration=2.418222036 podStartE2EDuration="2.418222036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:14:08.417306733 +0000 UTC m=+4245.815655358" watchObservedRunningTime="2026-10-31 14:14:08.418222036 +0000 UTC m=+4245.816570659"
Oct 31 14:14:08.549247 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:08.549209    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:09.325776 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:09.325736    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:10.041280 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:10.041242    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:10.777667 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:10.777631    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:11.640329 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:11.640290    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:12.435398 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:12.435355    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:13.341066 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:13.341033    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:14.036861 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:14.036831    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:14.371453 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:14.371372    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:14:14.771440 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:14.771398    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:15.629287 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:15.629207    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:16.434424 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:16.434390    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:17.029153 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:17.029111    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:17.541528 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:17.541490    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:18.334538 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:18.334502    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:18.739270 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:14:18.739231    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:14:19.128729 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:19.128647    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:19.318726 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:19.318690    4428 patch_prober.go:29] interesting pod/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" start-of-body=
Oct 31 14:14:19.318931 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:19.318751    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5" podUID="43a58059-e4ff-43d0-a312-2cb54fed07c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused"
Oct 31 14:14:19.362818 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:19.362789    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"
Oct 31 14:14:19.363312 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:19.362885    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"
Oct 31 14:14:19.434242 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:19.434149    4428 generic.go:356] "Generic (PLEG): container finished" podID="43a58059-e4ff-43d0-a312-2cb54fed07c3" containerID="1b506f95a081996f09a8c8dbfc21fc8852dc1de51f285dab4d610cc3a7231f1e" exitCode=0
Oct 31 14:14:19.434242 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:19.434224    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5" event={"ID":"43a58059-e4ff-43d0-a312-2cb54fed07c3","Type":"ContainerDied","Data":"1b506f95a081996f09a8c8dbfc21fc8852dc1de51f285dab4d610cc3a7231f1e"}
Oct 31 14:14:19.434761 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:19.434740    4428 scope.go:122] "RemoveContainer" containerID="1b506f95a081996f09a8c8dbfc21fc8852dc1de51f285dab4d610cc3a7231f1e"
Oct 31 14:14:19.436295 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:19.436259    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:14:19.436518 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:19.436493    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:14:19.436682 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:19.436548    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:14:19.932112 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:19.932065    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:20.441983 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:20.441940    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5" event={"ID":"43a58059-e4ff-43d0-a312-2cb54fed07c3","Type":"ContainerStarted","Data":"552781ccff939d07009509ead23028095b1074a0611c76daeede7d0834e9c3b3"}
Oct 31 14:14:20.509263 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:20.509225    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:20.509593 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:20.509564    4428 patch_prober.go:29] interesting pod/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" start-of-body=
Oct 31 14:14:20.509703 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:20.509638    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5" podUID="43a58059-e4ff-43d0-a312-2cb54fed07c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused"
Oct 31 14:14:20.637005 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:20.636966    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:20.802381 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:20.802340    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:21.316114 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:21.316071    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:21.471283 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:21.471251    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:21.509540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:21.509506    4428 patch_prober.go:29] interesting pod/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" start-of-body=
Oct 31 14:14:21.509741 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:21.509561    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5" podUID="43a58059-e4ff-43d0-a312-2cb54fed07c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused"
Oct 31 14:14:21.632280 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:21.632199    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:22.485951 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:22.485891    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:22.509635 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:22.509603    4428 patch_prober.go:29] interesting pod/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" start-of-body=
Oct 31 14:14:22.509759 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:22.509660    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5" podUID="43a58059-e4ff-43d0-a312-2cb54fed07c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused"
Oct 31 14:14:23.234820 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:23.234788    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:23.509815 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:23.509702    4428 patch_prober.go:29] interesting pod/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" start-of-body=
Oct 31 14:14:23.510365 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:23.509809    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5" podUID="43a58059-e4ff-43d0-a312-2cb54fed07c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused"
Oct 31 14:14:23.970956 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:23.968994    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:24.209705 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.209665    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"]
Oct 31 14:14:24.213837 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.213811    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.224912 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.224881    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a9dd7f\"/\"openshift-service-ca.crt\""
Oct 31 14:14:24.226142 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.226118    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a9dd7f\"/\"simulator-single-main-a9dd7f-kserve-self-signed-certs\""
Oct 31 14:14:24.226275 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.226171    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a9dd7f\"/\"kube-root-ca.crt\""
Oct 31 14:14:24.226342 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.226244    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a9dd7f\"/\"default-dockercfg-6ht56\""
Oct 31 14:14:24.229093 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.229067    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"]
Oct 31 14:14:24.338418 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.338382    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:24.390410 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.390368    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-dshm\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.390410 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.390415    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d527d4-c171-4372-81b0-15d7018c66d4-tls-certs\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.390706 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.390493    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-tmp-dir\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.390706 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.390538    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-model-cache\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.390706 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.390568    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-home\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.390706 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.390586    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp2xw\" (UniqueName: \"kubernetes.io/projected/33d527d4-c171-4372-81b0-15d7018c66d4-kube-api-access-fp2xw\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.491292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.491190    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-tmp-dir\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.491292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.491250    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-model-cache\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.491292 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.491280    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-home\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.491632 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.491298    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp2xw\" (UniqueName: \"kubernetes.io/projected/33d527d4-c171-4372-81b0-15d7018c66d4-kube-api-access-fp2xw\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.491632 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.491335    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-dshm\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.491632 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.491363    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d527d4-c171-4372-81b0-15d7018c66d4-tls-certs\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.491843 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.491687    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-model-cache\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.491843 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.491717    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-home\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.491843 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.491771    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-tmp-dir\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.493947 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.493924    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-dshm\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.494220 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.494202    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d527d4-c171-4372-81b0-15d7018c66d4-tls-certs\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.504858 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.504837    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp2xw\" (UniqueName: \"kubernetes.io/projected/33d527d4-c171-4372-81b0-15d7018c66d4-kube-api-access-fp2xw\") pod \"simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") " pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.509967 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.509944    4428 patch_prober.go:29] interesting pod/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" start-of-body=
Oct 31 14:14:24.510365 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.510012    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5" podUID="43a58059-e4ff-43d0-a312-2cb54fed07c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused"
Oct 31 14:14:24.536226 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.536195    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:24.858446 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.858355    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:24.877088 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:24.875972    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"]
Oct 31 14:14:25.500411 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:25.500370    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p" event={"ID":"33d527d4-c171-4372-81b0-15d7018c66d4","Type":"ContainerStarted","Data":"2eef7dd76cf07329d1d9ca2b31492721df55abdbecaf9d95294f92bf81a51b67"}
Oct 31 14:14:25.500603 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:25.500426    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p" event={"ID":"33d527d4-c171-4372-81b0-15d7018c66d4","Type":"ContainerStarted","Data":"f13710555e3fde92d049b94eea532a96fbd69acb072e1d3d4fb0a398058b57a9"}
Oct 31 14:14:25.523557 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:25.523496    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p" podStartSLOduration=1.523478911 podStartE2EDuration="1.523478911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:14:25.521538841 +0000 UTC m=+4262.919887466" watchObservedRunningTime="2026-10-31 14:14:25.523478911 +0000 UTC m=+4262.921827535"
Oct 31 14:14:25.536960 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:25.536922    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:25.585449 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:25.585413    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:25.586251 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:25.586221    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:25.589473 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:25.589444    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:14:25.662192 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:25.662144    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:25.917915 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:25.917802    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:26.077929 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:26.077897    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:26.560141 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:26.560105    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:27.353301 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:27.353262    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:28.128256 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:28.128215    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:28.901889 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:28.901834    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:29.776932 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:29.776886    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:30.625817 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:30.625784    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:31.030906 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:31.030857    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"]
Oct 31 14:14:31.035718 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:31.035691    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ngjxs"]
Oct 31 14:14:31.432311 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:31.432222    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:32.258283 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:32.258237    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:32.755190 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:32.755154    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba825ec5-dc91-4ffd-81cb-b91da48e83ba" path="/var/lib/kubelet/pods/ba825ec5-dc91-4ffd-81cb-b91da48e83ba/volumes"
Oct 31 14:14:33.152337 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:33.152244    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:33.944923 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:33.944835    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:34.537121 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:34.537084    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:34.551074 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:34.551041    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:34.562528 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:34.562500    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:14:34.762479 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:34.762442    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:35.567284 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:35.567257    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:36.356970 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:36.356937    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:36.378459 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:36.378432    4428 scope.go:122] "RemoveContainer" containerID="0ad9c867a4786524becd59c36ee228022ea158c7657e08a15be8b94f747cab8d"
Oct 31 14:14:36.388068 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:36.388044    4428 scope.go:122] "RemoveContainer" containerID="aff65769a450843c61a6f38f72cca608e2efc0ba9c6a2eba256032887439a4f9"
Oct 31 14:14:36.397238 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:36.397215    4428 scope.go:122] "RemoveContainer" containerID="fbafaf1d0ad90475336ed6ea502857b9ea1adeb48128feaf216cfc25aefdeabe"
Oct 31 14:14:37.150734 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:37.150675    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:37.945035 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:37.944999    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:38.772244 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:38.772200    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:39.546408 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:39.546369    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:40.346727 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:40.346675    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:41.319318 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:41.319281    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:42.025229 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:42.025185    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"]
Oct 31 14:14:42.028802 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:42.028770    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dqbng"]
Oct 31 14:14:42.139431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:42.139395    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:42.579716 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:42.579679    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:42.754759 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:42.754726    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9" path="/var/lib/kubelet/pods/5eb3c7b5-20fc-45c3-9b82-c2e3162f9fe9/volumes"
Oct 31 14:14:42.928509 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:42.928418    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:43.756487 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:43.756447    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:44.642014 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:44.641978    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:45.442764 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:45.442733    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:46.264400 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:46.264363    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:47.140969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:47.140933    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:47.921169 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:47.921130    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:48.651827 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:48.651780    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:48.740813 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:14:48.740768    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:14:49.509402 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:49.509353    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:50.240846 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:50.240807    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:51.045084 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:51.045052    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:51.920134 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:51.920097    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:53.227798 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:53.227761    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:53.579694 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:53.579594    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:54.374109 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:54.374066    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:55.255015 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:55.254980    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:56.051128 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:56.051094    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:56.862849 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:56.862801    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:57.649105 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:57.649071    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:58.437699 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:58.437664    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:14:59.328339 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:14:59.328305    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:00.136374 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:00.136337    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l"]
Oct 31 14:15:00.141682 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:00.141661    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l"
Oct 31 14:15:00.155256 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:00.155226    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:00.155503 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:00.155483    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:15:00.157824 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:00.157792    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l"]
Oct 31 14:15:00.273833 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:00.273789    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1649b61e-4314-4d04-b696-eebac5aaa77b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l\" (UID: \"1649b61e-4314-4d04-b696-eebac5aaa77b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l"
Oct 31 14:15:00.274073 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:00.274017    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tpf2\" (UniqueName: \"kubernetes.io/projected/1649b61e-4314-4d04-b696-eebac5aaa77b-kube-api-access-4tpf2\") pod \"maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l\" (UID: \"1649b61e-4314-4d04-b696-eebac5aaa77b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l"
Oct 31 14:15:00.375449 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:00.375407    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tpf2\" (UniqueName: \"kubernetes.io/projected/1649b61e-4314-4d04-b696-eebac5aaa77b-kube-api-access-4tpf2\") pod \"maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l\" (UID: \"1649b61e-4314-4d04-b696-eebac5aaa77b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l"
Oct 31 14:15:00.375909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:00.375479    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1649b61e-4314-4d04-b696-eebac5aaa77b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l\" (UID: \"1649b61e-4314-4d04-b696-eebac5aaa77b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l"
Oct 31 14:15:00.376174 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:00.376152    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1649b61e-4314-4d04-b696-eebac5aaa77b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l\" (UID: \"1649b61e-4314-4d04-b696-eebac5aaa77b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l"
Oct 31 14:15:00.388519 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:00.388446    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tpf2\" (UniqueName: \"kubernetes.io/projected/1649b61e-4314-4d04-b696-eebac5aaa77b-kube-api-access-4tpf2\") pod \"maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l\" (UID: \"1649b61e-4314-4d04-b696-eebac5aaa77b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l"
Oct 31 14:15:00.462795 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:00.462751    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l"
Oct 31 14:15:00.752452 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:15:00.752421    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1649b61e_4314_4d04_b696_eebac5aaa77b.slice/crio-29f7c39ea59e6b08ab6cf074ec553ac00db4d5f840c06a15e12d67ad7a9b31d9 WatchSource:0}: Error finding container 29f7c39ea59e6b08ab6cf074ec553ac00db4d5f840c06a15e12d67ad7a9b31d9: Status 404 returned error can't find the container with id 29f7c39ea59e6b08ab6cf074ec553ac00db4d5f840c06a15e12d67ad7a9b31d9
Oct 31 14:15:00.754618 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:00.754595    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l"]
Oct 31 14:15:00.950123 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:00.950090    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:01.698990 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:01.698954    4428 generic.go:356] "Generic (PLEG): container finished" podID="1649b61e-4314-4d04-b696-eebac5aaa77b" containerID="866dd900da3aa7168172be9f9763ceb417bc12ae7cb33100bb3f794798635f0c" exitCode=60
Oct 31 14:15:01.699430 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:01.699041    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l" event={"ID":"1649b61e-4314-4d04-b696-eebac5aaa77b","Type":"ContainerDied","Data":"866dd900da3aa7168172be9f9763ceb417bc12ae7cb33100bb3f794798635f0c"}
Oct 31 14:15:01.699430 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:01.699095    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l" event={"ID":"1649b61e-4314-4d04-b696-eebac5aaa77b","Type":"ContainerStarted","Data":"29f7c39ea59e6b08ab6cf074ec553ac00db4d5f840c06a15e12d67ad7a9b31d9"}
Oct 31 14:15:01.699430 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:01.699272    4428 scope.go:122] "RemoveContainer" containerID="866dd900da3aa7168172be9f9763ceb417bc12ae7cb33100bb3f794798635f0c"
Oct 31 14:15:01.857350 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:01.857254    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:02.706455 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:02.706352    4428 generic.go:356] "Generic (PLEG): container finished" podID="1649b61e-4314-4d04-b696-eebac5aaa77b" containerID="bc37e357b8b61c83a8faf6563d026fb34dbeba48d6f95e26117fdba23d5d15a9" exitCode=60
Oct 31 14:15:02.706455 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:02.706438    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l" event={"ID":"1649b61e-4314-4d04-b696-eebac5aaa77b","Type":"ContainerDied","Data":"bc37e357b8b61c83a8faf6563d026fb34dbeba48d6f95e26117fdba23d5d15a9"}
Oct 31 14:15:02.706973 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:02.706534    4428 scope.go:122] "RemoveContainer" containerID="866dd900da3aa7168172be9f9763ceb417bc12ae7cb33100bb3f794798635f0c"
Oct 31 14:15:02.706973 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:02.706768    4428 scope.go:122] "RemoveContainer" containerID="bc37e357b8b61c83a8faf6563d026fb34dbeba48d6f95e26117fdba23d5d15a9"
Oct 31 14:15:02.707120 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:15:02.707094    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l_odh-ai-gateway-infra(1649b61e-4314-4d04-b696-eebac5aaa77b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l" podUID="1649b61e-4314-4d04-b696-eebac5aaa77b"
Oct 31 14:15:02.749255 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:02.749214    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:03.542631 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:03.542596    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:03.713259 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:03.713229    4428 scope.go:122] "RemoveContainer" containerID="bc37e357b8b61c83a8faf6563d026fb34dbeba48d6f95e26117fdba23d5d15a9"
Oct 31 14:15:03.713755 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:15:03.713495    4428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l_odh-ai-gateway-infra(1649b61e-4314-4d04-b696-eebac5aaa77b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l" podUID="1649b61e-4314-4d04-b696-eebac5aaa77b"
Oct 31 14:15:03.815099 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:03.815007    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:04.586434 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:04.586391    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:05.426548 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:05.426514    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:06.241427 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:06.241398    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:07.191702 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:07.191672    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:07.937050 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:07.937012    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:08.816322 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:08.816290    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:08.816835 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:08.816815    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"]
Oct 31 14:15:08.973042 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:08.973006    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:09.561566 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:09.561533    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:10.250024 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:10.249986    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:11.045756 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:11.045727    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:11.843958 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:11.843920    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:12.656810 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:12.656777    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:13.559336 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:13.559298    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:13.748170 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:13.748139    4428 scope.go:122] "RemoveContainer" containerID="bc37e357b8b61c83a8faf6563d026fb34dbeba48d6f95e26117fdba23d5d15a9"
Oct 31 14:15:13.960437 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:13.960410    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"]
Oct 31 14:15:13.976021 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:13.975992    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"]
Oct 31 14:15:14.114187 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.114151    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:14.177172 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.177133    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"]
Oct 31 14:15:14.177442 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.177416    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz" podUID="e37f1ca7-f48b-43be-9bd5-ba38e394653c" containerName="maas-api" containerID="cri-o://13a22cf4624aa531151e848decb762cb37d3a281c8f5e0c58148c3c54c0010c9" gracePeriod=30
Oct 31 14:15:14.222475 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.222399    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l"]
Oct 31 14:15:14.278795 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.278753    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:14.360843 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.360605    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz" podUID="e37f1ca7-f48b-43be-9bd5-ba38e394653c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.1.84:8443/health\": dial tcp 10.134.1.84:8443: connect: connection refused"
Oct 31 14:15:14.448358 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.448302    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:14.462255 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.462231    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:15:14.517447 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.517354    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e37f1ca7-f48b-43be-9bd5-ba38e394653c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e37f1ca7-f48b-43be-9bd5-ba38e394653c-maas-api-tls\") pod \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\" (UID: \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\") "
Oct 31 14:15:14.517652 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.517455    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e37f1ca7-f48b-43be-9bd5-ba38e394653c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e37f1ca7-f48b-43be-9bd5-ba38e394653c-openshift-service-ca\") pod \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\" (UID: \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\") "
Oct 31 14:15:14.517652 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.517491    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37f1ca7-f48b-43be-9bd5-ba38e394653c-kube-api-access-cbsd2\" (UniqueName: \"kubernetes.io/projected/e37f1ca7-f48b-43be-9bd5-ba38e394653c-kube-api-access-cbsd2\") pod \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\" (UID: \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\") "
Oct 31 14:15:14.517652 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.517570    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e37f1ca7-f48b-43be-9bd5-ba38e394653c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e37f1ca7-f48b-43be-9bd5-ba38e394653c-metrics-certs\") pod \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\" (UID: \"e37f1ca7-f48b-43be-9bd5-ba38e394653c\") "
Oct 31 14:15:14.517922 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.517851    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e37f1ca7-f48b-43be-9bd5-ba38e394653c-openshift-service-ca" pod "e37f1ca7-f48b-43be-9bd5-ba38e394653c" (UID: "e37f1ca7-f48b-43be-9bd5-ba38e394653c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:15:14.520368 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.520340    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37f1ca7-f48b-43be-9bd5-ba38e394653c-kube-api-access-cbsd2" pod "e37f1ca7-f48b-43be-9bd5-ba38e394653c" (UID: "e37f1ca7-f48b-43be-9bd5-ba38e394653c"). InnerVolumeSpecName "kube-api-access-cbsd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:15:14.520368 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.520347    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e37f1ca7-f48b-43be-9bd5-ba38e394653c-maas-api-tls" pod "e37f1ca7-f48b-43be-9bd5-ba38e394653c" (UID: "e37f1ca7-f48b-43be-9bd5-ba38e394653c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:15:14.520558 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.520382    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e37f1ca7-f48b-43be-9bd5-ba38e394653c-metrics-certs" pod "e37f1ca7-f48b-43be-9bd5-ba38e394653c" (UID: "e37f1ca7-f48b-43be-9bd5-ba38e394653c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:15:14.618471 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.618434    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e37f1ca7-f48b-43be-9bd5-ba38e394653c-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:14.618471 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.618465    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbsd2\" (UniqueName: \"kubernetes.io/projected/e37f1ca7-f48b-43be-9bd5-ba38e394653c-kube-api-access-cbsd2\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:14.618471 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.618476    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e37f1ca7-f48b-43be-9bd5-ba38e394653c-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:14.618980 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.618486    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e37f1ca7-f48b-43be-9bd5-ba38e394653c-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:14.775310 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.775273    4428 generic.go:356] "Generic (PLEG): container finished" podID="1649b61e-4314-4d04-b696-eebac5aaa77b" containerID="7177da8dcc068f14564a7fd3802a0c58ed6693fb94eb451e0501f3c6b47f99fc" exitCode=6
Oct 31 14:15:14.775486 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.775353    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l" event={"ID":"1649b61e-4314-4d04-b696-eebac5aaa77b","Type":"ContainerDied","Data":"7177da8dcc068f14564a7fd3802a0c58ed6693fb94eb451e0501f3c6b47f99fc"}
Oct 31 14:15:14.775486 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.775420    4428 scope.go:122] "RemoveContainer" containerID="bc37e357b8b61c83a8faf6563d026fb34dbeba48d6f95e26117fdba23d5d15a9"
Oct 31 14:15:14.776931 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.776908    4428 generic.go:356] "Generic (PLEG): container finished" podID="e37f1ca7-f48b-43be-9bd5-ba38e394653c" containerID="13a22cf4624aa531151e848decb762cb37d3a281c8f5e0c58148c3c54c0010c9" exitCode=0
Oct 31 14:15:14.777039 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.776964    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"
Oct 31 14:15:14.777039 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.776990    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz" event={"ID":"e37f1ca7-f48b-43be-9bd5-ba38e394653c","Type":"ContainerDied","Data":"13a22cf4624aa531151e848decb762cb37d3a281c8f5e0c58148c3c54c0010c9"}
Oct 31 14:15:14.777141 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.777043    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz" event={"ID":"e37f1ca7-f48b-43be-9bd5-ba38e394653c","Type":"ContainerDied","Data":"3b5ea34878f88a9886c231e1f631edfb5664696a9dc7d9b89e1a4e25724f9e39"}
Oct 31 14:15:14.789709 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.789691    4428 scope.go:122] "RemoveContainer" containerID="13a22cf4624aa531151e848decb762cb37d3a281c8f5e0c58148c3c54c0010c9"
Oct 31 14:15:14.799731 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.799701    4428 scope.go:122] "RemoveContainer" containerID="13a22cf4624aa531151e848decb762cb37d3a281c8f5e0c58148c3c54c0010c9"
Oct 31 14:15:14.800087 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:15:14.800058    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a22cf4624aa531151e848decb762cb37d3a281c8f5e0c58148c3c54c0010c9\": container with ID starting with 13a22cf4624aa531151e848decb762cb37d3a281c8f5e0c58148c3c54c0010c9 not found: ID does not exist" containerID="13a22cf4624aa531151e848decb762cb37d3a281c8f5e0c58148c3c54c0010c9"
Oct 31 14:15:14.800181 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.800104    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a22cf4624aa531151e848decb762cb37d3a281c8f5e0c58148c3c54c0010c9"} err="failed to get container status \"13a22cf4624aa531151e848decb762cb37d3a281c8f5e0c58148c3c54c0010c9\": rpc error: code = NotFound desc = could not find container \"13a22cf4624aa531151e848decb762cb37d3a281c8f5e0c58148c3c54c0010c9\": container with ID starting with 13a22cf4624aa531151e848decb762cb37d3a281c8f5e0c58148c3c54c0010c9 not found: ID does not exist"
Oct 31 14:15:14.845696 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.845656    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"]
Oct 31 14:15:14.849333 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.849306    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dd7f-766b5c945-zq4wz"]
Oct 31 14:15:14.947346 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:14.947316    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l"
Oct 31 14:15:15.021531 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:15.021444    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1649b61e-4314-4d04-b696-eebac5aaa77b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1649b61e-4314-4d04-b696-eebac5aaa77b-maas-api-service-ca\") pod \"1649b61e-4314-4d04-b696-eebac5aaa77b\" (UID: \"1649b61e-4314-4d04-b696-eebac5aaa77b\") "
Oct 31 14:15:15.021531 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:15.021508    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1649b61e-4314-4d04-b696-eebac5aaa77b-kube-api-access-4tpf2\" (UniqueName: \"kubernetes.io/projected/1649b61e-4314-4d04-b696-eebac5aaa77b-kube-api-access-4tpf2\") pod \"1649b61e-4314-4d04-b696-eebac5aaa77b\" (UID: \"1649b61e-4314-4d04-b696-eebac5aaa77b\") "
Oct 31 14:15:15.021821 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:15.021798    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1649b61e-4314-4d04-b696-eebac5aaa77b-maas-api-service-ca" pod "1649b61e-4314-4d04-b696-eebac5aaa77b" (UID: "1649b61e-4314-4d04-b696-eebac5aaa77b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:15:15.023901 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:15.023879    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1649b61e-4314-4d04-b696-eebac5aaa77b-kube-api-access-4tpf2" pod "1649b61e-4314-4d04-b696-eebac5aaa77b" (UID: "1649b61e-4314-4d04-b696-eebac5aaa77b"). InnerVolumeSpecName "kube-api-access-4tpf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:15:15.122348 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:15.122307    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1649b61e-4314-4d04-b696-eebac5aaa77b-maas-api-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:15.122348 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:15.122347    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tpf2\" (UniqueName: \"kubernetes.io/projected/1649b61e-4314-4d04-b696-eebac5aaa77b-kube-api-access-4tpf2\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:15.216559 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:15.216520    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:15.388428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:15.388340    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:15.780415 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:15.780386    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:15.784364 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:15.784335    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l"
Oct 31 14:15:15.784364 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:15.784339    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l" event={"ID":"1649b61e-4314-4d04-b696-eebac5aaa77b","Type":"ContainerDied","Data":"29f7c39ea59e6b08ab6cf074ec553ac00db4d5f840c06a15e12d67ad7a9b31d9"}
Oct 31 14:15:15.784620 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:15.784405    4428 scope.go:122] "RemoveContainer" containerID="7177da8dcc068f14564a7fd3802a0c58ed6693fb94eb451e0501f3c6b47f99fc"
Oct 31 14:15:15.811547 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:15.811501    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l"]
Oct 31 14:15:15.816283 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:15.816256    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-a9dd7f-2989093l7g2l"]
Oct 31 14:15:16.478944 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:16.478910    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:16.754299 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:16.754204    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1649b61e-4314-4d04-b696-eebac5aaa77b" path="/var/lib/kubelet/pods/1649b61e-4314-4d04-b696-eebac5aaa77b/volumes"
Oct 31 14:15:16.754663 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:16.754639    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e37f1ca7-f48b-43be-9bd5-ba38e394653c" path="/var/lib/kubelet/pods/e37f1ca7-f48b-43be-9bd5-ba38e394653c/volumes"
Oct 31 14:15:17.287437 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:17.287402    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:18.743034 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:15:18.742997    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:15:18.961243 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:18.961108    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s" podUID="3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe" containerName="payload-processing" containerID="cri-o://1e9949005068167b37a79bde2324f9a7546addb54a398e45484b922a74e70472" gracePeriod=25
Oct 31 14:15:18.977180 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:18.977128    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf" podUID="21912d67-a608-4417-9191-df1dcc8d958c" containerName="payload-pre-processing" containerID="cri-o://e15c8732e20f1bfffcb329b1c104f1692c77be6852cab8fdc45d2b266f16228c" gracePeriod=25
Oct 31 14:15:19.361765 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:19.361715    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf" podUID="21912d67-a608-4417-9191-df1dcc8d958c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:15:19.361765 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:19.361748    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s" podUID="3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:15:22.215641 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:22.215592    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:22.373536 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:22.373498    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:22.486497 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:22.486322    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"]
Oct 31 14:15:22.486699 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:22.486634    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5" podUID="43a58059-e4ff-43d0-a312-2cb54fed07c3" containerName="istio-proxy" containerID="cri-o://552781ccff939d07009509ead23028095b1074a0611c76daeede7d0834e9c3b3" gracePeriod=30
Oct 31 14:15:22.592108 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:22.592018    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:23.838016 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:23.837971    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p" podUID="33d527d4-c171-4372-81b0-15d7018c66d4" containerName="main" containerID="cri-o://2eef7dd76cf07329d1d9ca2b31492721df55abdbecaf9d95294f92bf81a51b67" gracePeriod=45
Oct 31 14:15:24.098559 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.098489    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:15:24.106434 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.106402    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-model-cache\") pod \"33d527d4-c171-4372-81b0-15d7018c66d4\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") "
Oct 31 14:15:24.106607 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.106448    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-dshm\") pod \"33d527d4-c171-4372-81b0-15d7018c66d4\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") "
Oct 31 14:15:24.106607 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.106478    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d527d4-c171-4372-81b0-15d7018c66d4-kube-api-access-fp2xw\" (UniqueName: \"kubernetes.io/projected/33d527d4-c171-4372-81b0-15d7018c66d4-kube-api-access-fp2xw\") pod \"33d527d4-c171-4372-81b0-15d7018c66d4\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") "
Oct 31 14:15:24.106607 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.106569    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33d527d4-c171-4372-81b0-15d7018c66d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/33d527d4-c171-4372-81b0-15d7018c66d4-tls-certs\") pod \"33d527d4-c171-4372-81b0-15d7018c66d4\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") "
Oct 31 14:15:24.106607 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.106602    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-home\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-home\") pod \"33d527d4-c171-4372-81b0-15d7018c66d4\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") "
Oct 31 14:15:24.106900 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.106631    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-tmp-dir\") pod \"33d527d4-c171-4372-81b0-15d7018c66d4\" (UID: \"33d527d4-c171-4372-81b0-15d7018c66d4\") "
Oct 31 14:15:24.106900 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.106657    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-model-cache" pod "33d527d4-c171-4372-81b0-15d7018c66d4" (UID: "33d527d4-c171-4372-81b0-15d7018c66d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:15:24.107018 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.106896    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-home" pod "33d527d4-c171-4372-81b0-15d7018c66d4" (UID: "33d527d4-c171-4372-81b0-15d7018c66d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:15:24.107018 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.106982    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:24.107018 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.106989    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-tmp-dir" pod "33d527d4-c171-4372-81b0-15d7018c66d4" (UID: "33d527d4-c171-4372-81b0-15d7018c66d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:15:24.107018 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.107004    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:24.108833 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.108807    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-dshm" pod "33d527d4-c171-4372-81b0-15d7018c66d4" (UID: "33d527d4-c171-4372-81b0-15d7018c66d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:15:24.109340 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.109320    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33d527d4-c171-4372-81b0-15d7018c66d4-tls-certs" pod "33d527d4-c171-4372-81b0-15d7018c66d4" (UID: "33d527d4-c171-4372-81b0-15d7018c66d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:15:24.109433 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.109356    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d527d4-c171-4372-81b0-15d7018c66d4-kube-api-access-fp2xw" pod "33d527d4-c171-4372-81b0-15d7018c66d4" (UID: "33d527d4-c171-4372-81b0-15d7018c66d4"). InnerVolumeSpecName "kube-api-access-fp2xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:15:24.208147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.208112    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:24.208147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.208140    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp2xw\" (UniqueName: \"kubernetes.io/projected/33d527d4-c171-4372-81b0-15d7018c66d4-kube-api-access-fp2xw\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:24.208147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.208150    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d527d4-c171-4372-81b0-15d7018c66d4-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:24.208421 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.208180    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d527d4-c171-4372-81b0-15d7018c66d4-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:24.836644 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.836610    4428 generic.go:356] "Generic (PLEG): container finished" podID="33d527d4-c171-4372-81b0-15d7018c66d4" containerID="2eef7dd76cf07329d1d9ca2b31492721df55abdbecaf9d95294f92bf81a51b67" exitCode=0
Oct 31 14:15:24.836849 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.836657    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"
Oct 31 14:15:24.836849 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.836696    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p" event={"ID":"33d527d4-c171-4372-81b0-15d7018c66d4","Type":"ContainerDied","Data":"2eef7dd76cf07329d1d9ca2b31492721df55abdbecaf9d95294f92bf81a51b67"}
Oct 31 14:15:24.836849 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.836747    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p" event={"ID":"33d527d4-c171-4372-81b0-15d7018c66d4","Type":"ContainerDied","Data":"f13710555e3fde92d049b94eea532a96fbd69acb072e1d3d4fb0a398058b57a9"}
Oct 31 14:15:24.836849 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.836769    4428 scope.go:122] "RemoveContainer" containerID="2eef7dd76cf07329d1d9ca2b31492721df55abdbecaf9d95294f92bf81a51b67"
Oct 31 14:15:24.848852 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.848623    4428 scope.go:122] "RemoveContainer" containerID="2eef7dd76cf07329d1d9ca2b31492721df55abdbecaf9d95294f92bf81a51b67"
Oct 31 14:15:24.849133 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:15:24.848949    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eef7dd76cf07329d1d9ca2b31492721df55abdbecaf9d95294f92bf81a51b67\": container with ID starting with 2eef7dd76cf07329d1d9ca2b31492721df55abdbecaf9d95294f92bf81a51b67 not found: ID does not exist" containerID="2eef7dd76cf07329d1d9ca2b31492721df55abdbecaf9d95294f92bf81a51b67"
Oct 31 14:15:24.849133 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.848976    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eef7dd76cf07329d1d9ca2b31492721df55abdbecaf9d95294f92bf81a51b67"} err="failed to get container status \"2eef7dd76cf07329d1d9ca2b31492721df55abdbecaf9d95294f92bf81a51b67\": rpc error: code = NotFound desc = could not find container \"2eef7dd76cf07329d1d9ca2b31492721df55abdbecaf9d95294f92bf81a51b67\": container with ID starting with 2eef7dd76cf07329d1d9ca2b31492721df55abdbecaf9d95294f92bf81a51b67 not found: ID does not exist"
Oct 31 14:15:24.856200 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.856166    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"]
Oct 31 14:15:24.861778 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:24.861748    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-a9dd7f/simulator-single-main-a9dd7f-kserve-5f4f84b545-6942p"]
Oct 31 14:15:25.588398 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:25.588355    4428 patch_prober.go:29] interesting pod/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:15:25.588589 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:25.588439    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5" podUID="43a58059-e4ff-43d0-a312-2cb54fed07c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:15:26.754288 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:26.754254    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33d527d4-c171-4372-81b0-15d7018c66d4" path="/var/lib/kubelet/pods/33d527d4-c171-4372-81b0-15d7018c66d4/volumes"
Oct 31 14:15:27.855449 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.855353    4428 generic.go:356] "Generic (PLEG): container finished" podID="43a58059-e4ff-43d0-a312-2cb54fed07c3" containerID="552781ccff939d07009509ead23028095b1074a0611c76daeede7d0834e9c3b3" exitCode=0
Oct 31 14:15:27.855449 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.855435    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5" event={"ID":"43a58059-e4ff-43d0-a312-2cb54fed07c3","Type":"ContainerDied","Data":"552781ccff939d07009509ead23028095b1074a0611c76daeede7d0834e9c3b3"}
Oct 31 14:15:27.855971 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.855514    4428 scope.go:122] "RemoveContainer" containerID="1b506f95a081996f09a8c8dbfc21fc8852dc1de51f285dab4d610cc3a7231f1e"
Oct 31 14:15:27.997205 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.997168    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"]
Oct 31 14:15:27.997777 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.997749    4428 cpu_manager.go:414] "containerMap: removing container" podUID="1649b61e-4314-4d04-b696-eebac5aaa77b" containerName="cleanup"
Oct 31 14:15:27.997777 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.997774    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1649b61e-4314-4d04-b696-eebac5aaa77b" containerName="cleanup"
Oct 31 14:15:27.998048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.997810    4428 cpu_manager.go:414] "containerMap: removing container" podUID="1649b61e-4314-4d04-b696-eebac5aaa77b" containerName="cleanup"
Oct 31 14:15:27.998048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.997818    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1649b61e-4314-4d04-b696-eebac5aaa77b" containerName="cleanup"
Oct 31 14:15:27.998048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.997830    4428 cpu_manager.go:414] "containerMap: removing container" podUID="1649b61e-4314-4d04-b696-eebac5aaa77b" containerName="cleanup"
Oct 31 14:15:27.998048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.997839    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1649b61e-4314-4d04-b696-eebac5aaa77b" containerName="cleanup"
Oct 31 14:15:27.998048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.997862    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e37f1ca7-f48b-43be-9bd5-ba38e394653c" containerName="maas-api"
Oct 31 14:15:27.998048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.997886    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e37f1ca7-f48b-43be-9bd5-ba38e394653c" containerName="maas-api"
Oct 31 14:15:27.998048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.997902    4428 cpu_manager.go:414] "containerMap: removing container" podUID="33d527d4-c171-4372-81b0-15d7018c66d4" containerName="main"
Oct 31 14:15:27.998048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.997909    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33d527d4-c171-4372-81b0-15d7018c66d4" containerName="main"
Oct 31 14:15:27.998048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.998018    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1649b61e-4314-4d04-b696-eebac5aaa77b" containerName="cleanup"
Oct 31 14:15:27.998048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.998030    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1649b61e-4314-4d04-b696-eebac5aaa77b" containerName="cleanup"
Oct 31 14:15:27.998048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.998044    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e37f1ca7-f48b-43be-9bd5-ba38e394653c" containerName="maas-api"
Oct 31 14:15:27.998048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:27.998054    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="33d527d4-c171-4372-81b0-15d7018c66d4" containerName="main"
Oct 31 14:15:28.002626 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.002602    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.010557 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.009885    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-9e257301-kserve-self-signed-certs\""
Oct 31 14:15:28.011586 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.011560    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"]
Oct 31 14:15:28.039620 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.039582    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-home\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.039825 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.039655    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98lrt\" (UniqueName: \"kubernetes.io/projected/5f404e76-c160-441d-8ac3-0a0a36fba326-kube-api-access-98lrt\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.039825 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.039784    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-model-cache\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.039984 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.039948    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-tmp-dir\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.040061 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.040042    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f404e76-c160-441d-8ac3-0a0a36fba326-tls-certs\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.040138 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.040099    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-dshm\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.126633 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.126531    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:28.141649 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.141605    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-home\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.142021 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.141686    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98lrt\" (UniqueName: \"kubernetes.io/projected/5f404e76-c160-441d-8ac3-0a0a36fba326-kube-api-access-98lrt\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.142021 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.141787    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-model-cache\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.142021 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.141903    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-tmp-dir\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.142021 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.141970    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f404e76-c160-441d-8ac3-0a0a36fba326-tls-certs\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.142021 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.142009    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-dshm\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.143148 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.142738    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-home\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.143148 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.142813    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-model-cache\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.143148 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.143053    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-tmp-dir\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.149656 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.149620    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-dshm\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.151020 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.150958    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f404e76-c160-441d-8ac3-0a0a36fba326-tls-certs\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.156858 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.156834    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98lrt\" (UniqueName: \"kubernetes.io/projected/5f404e76-c160-441d-8ac3-0a0a36fba326-kube-api-access-98lrt\") pod \"e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") " pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.181388 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.181361    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:15:28.243248 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.243203    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-workload-certs\") pod \"43a58059-e4ff-43d0-a312-2cb54fed07c3\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") "
Oct 31 14:15:28.243465 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.243274    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-credential-socket\") pod \"43a58059-e4ff-43d0-a312-2cb54fed07c3\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") "
Oct 31 14:15:28.243465 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.243317    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a58059-e4ff-43d0-a312-2cb54fed07c3-kube-api-access-9v9ns\" (UniqueName: \"kubernetes.io/projected/43a58059-e4ff-43d0-a312-2cb54fed07c3-kube-api-access-9v9ns\") pod \"43a58059-e4ff-43d0-a312-2cb54fed07c3\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") "
Oct 31 14:15:28.243465 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.243372    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-envoy\") pod \"43a58059-e4ff-43d0-a312-2cb54fed07c3\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") "
Oct 31 14:15:28.243465 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.243406    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-token\" (UniqueName: \"kubernetes.io/projected/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-token\") pod \"43a58059-e4ff-43d0-a312-2cb54fed07c3\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") "
Oct 31 14:15:28.243465 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.243461    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-data\") pod \"43a58059-e4ff-43d0-a312-2cb54fed07c3\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") "
Oct 31 14:15:28.243861 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.243485    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a58059-e4ff-43d0-a312-2cb54fed07c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a58059-e4ff-43d0-a312-2cb54fed07c3-istiod-ca-cert\") pod \"43a58059-e4ff-43d0-a312-2cb54fed07c3\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") "
Oct 31 14:15:28.243861 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.243528    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-workload-socket\") pod \"43a58059-e4ff-43d0-a312-2cb54fed07c3\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") "
Oct 31 14:15:28.243861 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.243571    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-podinfo\") pod \"43a58059-e4ff-43d0-a312-2cb54fed07c3\" (UID: \"43a58059-e4ff-43d0-a312-2cb54fed07c3\") "
Oct 31 14:15:28.243861 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.243605    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-workload-certs" pod "43a58059-e4ff-43d0-a312-2cb54fed07c3" (UID: "43a58059-e4ff-43d0-a312-2cb54fed07c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:15:28.243861 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.243622    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-credential-socket" pod "43a58059-e4ff-43d0-a312-2cb54fed07c3" (UID: "43a58059-e4ff-43d0-a312-2cb54fed07c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:15:28.243861 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.243834    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-data" pod "43a58059-e4ff-43d0-a312-2cb54fed07c3" (UID: "43a58059-e4ff-43d0-a312-2cb54fed07c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:15:28.244261 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.244010    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:28.244261 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.244034    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:28.244261 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.244049    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:28.244261 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.244193    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-workload-socket" pod "43a58059-e4ff-43d0-a312-2cb54fed07c3" (UID: "43a58059-e4ff-43d0-a312-2cb54fed07c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:15:28.244499 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.244361    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a58059-e4ff-43d0-a312-2cb54fed07c3-istiod-ca-cert" pod "43a58059-e4ff-43d0-a312-2cb54fed07c3" (UID: "43a58059-e4ff-43d0-a312-2cb54fed07c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:15:28.246125 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.246085    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a58059-e4ff-43d0-a312-2cb54fed07c3-kube-api-access-9v9ns" pod "43a58059-e4ff-43d0-a312-2cb54fed07c3" (UID: "43a58059-e4ff-43d0-a312-2cb54fed07c3"). InnerVolumeSpecName "kube-api-access-9v9ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:15:28.246395 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.246375    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-token" pod "43a58059-e4ff-43d0-a312-2cb54fed07c3" (UID: "43a58059-e4ff-43d0-a312-2cb54fed07c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:15:28.246791 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.246767    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-podinfo" pod "43a58059-e4ff-43d0-a312-2cb54fed07c3" (UID: "43a58059-e4ff-43d0-a312-2cb54fed07c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:15:28.246929 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.246801    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-envoy" pod "43a58059-e4ff-43d0-a312-2cb54fed07c3" (UID: "43a58059-e4ff-43d0-a312-2cb54fed07c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:15:28.323171 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.323129    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:28.344574 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.344545    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v9ns\" (UniqueName: \"kubernetes.io/projected/43a58059-e4ff-43d0-a312-2cb54fed07c3-kube-api-access-9v9ns\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:28.344574 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.344574    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:28.344747 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.344586    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:28.344747 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.344597    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a58059-e4ff-43d0-a312-2cb54fed07c3-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:28.344747 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.344606    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58059-e4ff-43d0-a312-2cb54fed07c3-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:28.344747 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.344615    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a58059-e4ff-43d0-a312-2cb54fed07c3-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:28.603897 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.603849    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"]
Oct 31 14:15:28.607257 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:15:28.607223    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f404e76_c160_441d_8ac3_0a0a36fba326.slice/crio-31d5d37ff96aea8507a6c6580cfb520f597ecdbd5e8d2ca102c11fb0b5c5f2d9 WatchSource:0}: Error finding container 31d5d37ff96aea8507a6c6580cfb520f597ecdbd5e8d2ca102c11fb0b5c5f2d9: Status 404 returned error can't find the container with id 31d5d37ff96aea8507a6c6580cfb520f597ecdbd5e8d2ca102c11fb0b5c5f2d9
Oct 31 14:15:28.862422 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.862380    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5" event={"ID":"43a58059-e4ff-43d0-a312-2cb54fed07c3","Type":"ContainerDied","Data":"fe10e17e0278374ad59ffc6136e8997f9810f5393e30fae90cb0009bb6f20deb"}
Oct 31 14:15:28.862422 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.862399    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"
Oct 31 14:15:28.863113 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.862489    4428 scope.go:122] "RemoveContainer" containerID="552781ccff939d07009509ead23028095b1074a0611c76daeede7d0834e9c3b3"
Oct 31 14:15:28.864665 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.864635    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt" event={"ID":"5f404e76-c160-441d-8ac3-0a0a36fba326","Type":"ContainerStarted","Data":"a631a7dd0a0cc646f9c50aec8d44b9372acda6f8aa40db94d6383043134cce51"}
Oct 31 14:15:28.864800 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.864675    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt" event={"ID":"5f404e76-c160-441d-8ac3-0a0a36fba326","Type":"ContainerStarted","Data":"31d5d37ff96aea8507a6c6580cfb520f597ecdbd5e8d2ca102c11fb0b5c5f2d9"}
Oct 31 14:15:28.889763 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.889732    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"]
Oct 31 14:15:28.896975 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.896944    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a9dd7f-openshift-default-7667cff5dfw4mw5"]
Oct 31 14:15:28.919859 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:28.919754    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt" podStartSLOduration=1.9197372430000001 podStartE2EDuration="1.919737243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:15:28.916915944 +0000 UTC m=+4326.315264572" watchObservedRunningTime="2026-10-31 14:15:28.919737243 +0000 UTC m=+4326.318085937"
Oct 31 14:15:29.323684 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:29.323654    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:29.361227 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:29.361195    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s" podUID="3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:15:29.361442 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:29.361200    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf" podUID="21912d67-a608-4417-9191-df1dcc8d958c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 14:15:30.756531 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:30.756489    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a58059-e4ff-43d0-a312-2cb54fed07c3" path="/var/lib/kubelet/pods/43a58059-e4ff-43d0-a312-2cb54fed07c3/volumes"
Oct 31 14:15:36.425814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:36.425783    4428 scope.go:122] "RemoveContainer" containerID="4c83dff49f65f9a67a6ce50444f5cfbec6a130f2b006457df5051da37f5d2cd1"
Oct 31 14:15:36.434931 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:36.434910    4428 scope.go:122] "RemoveContainer" containerID="654e3a0d1b61fce3bd8723eea985a862e21afc7bcdc6dee2559082d433385b8d"
Oct 31 14:15:36.443964 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:36.443935    4428 scope.go:122] "RemoveContainer" containerID="8053a305f7efeea7e5d061b53a59e7ce1e264dbb4003fb310b391b2955b6a415"
Oct 31 14:15:38.324331 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:38.324296    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:38.336983 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:38.336954    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:38.965769 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:38.965737    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:15:39.223243 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.223178    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"
Oct 31 14:15:39.267711 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.267680    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"
Oct 31 14:15:39.313981 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.313946    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe-plugins-config-volume\") pod \"3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe\" (UID: \"3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe\") "
Oct 31 14:15:39.314169 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.314021    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe-kube-api-access-rtdzq\" (UniqueName: \"kubernetes.io/projected/3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe-kube-api-access-rtdzq\") pod \"3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe\" (UID: \"3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe\") "
Oct 31 14:15:39.314169 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.314055    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21912d67-a608-4417-9191-df1dcc8d958c-kube-api-access-7kkd7\" (UniqueName: \"kubernetes.io/projected/21912d67-a608-4417-9191-df1dcc8d958c-kube-api-access-7kkd7\") pod \"21912d67-a608-4417-9191-df1dcc8d958c\" (UID: \"21912d67-a608-4417-9191-df1dcc8d958c\") "
Oct 31 14:15:39.314323 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.314240    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21912d67-a608-4417-9191-df1dcc8d958c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21912d67-a608-4417-9191-df1dcc8d958c-plugins-config-volume\") pod \"21912d67-a608-4417-9191-df1dcc8d958c\" (UID: \"21912d67-a608-4417-9191-df1dcc8d958c\") "
Oct 31 14:15:39.314386 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.314362    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe-plugins-config-volume" pod "3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe" (UID: "3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:15:39.314615 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.314592    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21912d67-a608-4417-9191-df1dcc8d958c-plugins-config-volume" pod "21912d67-a608-4417-9191-df1dcc8d958c" (UID: "21912d67-a608-4417-9191-df1dcc8d958c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:15:39.314985 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.314967    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:39.315081 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.314990    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21912d67-a608-4417-9191-df1dcc8d958c-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:39.317042 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.317020    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe-kube-api-access-rtdzq" pod "3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe" (UID: "3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe"). InnerVolumeSpecName "kube-api-access-rtdzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:15:39.317123 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.317050    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21912d67-a608-4417-9191-df1dcc8d958c-kube-api-access-7kkd7" pod "21912d67-a608-4417-9191-df1dcc8d958c" (UID: "21912d67-a608-4417-9191-df1dcc8d958c"). InnerVolumeSpecName "kube-api-access-7kkd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:15:39.416008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.415973    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtdzq\" (UniqueName: \"kubernetes.io/projected/3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe-kube-api-access-rtdzq\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:39.416008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.416003    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kkd7\" (UniqueName: \"kubernetes.io/projected/21912d67-a608-4417-9191-df1dcc8d958c-kube-api-access-7kkd7\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:15:39.961555 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.961510    4428 generic.go:356] "Generic (PLEG): container finished" podID="3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe" containerID="1e9949005068167b37a79bde2324f9a7546addb54a398e45484b922a74e70472" exitCode=0
Oct 31 14:15:39.961555 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.961557    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"
Oct 31 14:15:39.961849 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.961560    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s" event={"ID":"3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe","Type":"ContainerDied","Data":"1e9949005068167b37a79bde2324f9a7546addb54a398e45484b922a74e70472"}
Oct 31 14:15:39.961849 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.961629    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s" event={"ID":"3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe","Type":"ContainerDied","Data":"472d1eebcfed1eb67304a9cc2777c87c603418f6f831274bb6e108a44931468f"}
Oct 31 14:15:39.961849 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.961660    4428 scope.go:122] "RemoveContainer" containerID="1e9949005068167b37a79bde2324f9a7546addb54a398e45484b922a74e70472"
Oct 31 14:15:39.963296 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.963238    4428 generic.go:356] "Generic (PLEG): container finished" podID="21912d67-a608-4417-9191-df1dcc8d958c" containerID="e15c8732e20f1bfffcb329b1c104f1692c77be6852cab8fdc45d2b266f16228c" exitCode=0
Oct 31 14:15:39.963296 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.963308    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"
Oct 31 14:15:39.963296 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.963329    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf" event={"ID":"21912d67-a608-4417-9191-df1dcc8d958c","Type":"ContainerDied","Data":"e15c8732e20f1bfffcb329b1c104f1692c77be6852cab8fdc45d2b266f16228c"}
Oct 31 14:15:39.963647 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.963390    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf" event={"ID":"21912d67-a608-4417-9191-df1dcc8d958c","Type":"ContainerDied","Data":"25e8c97407b1cccd63e9482b983f781c5a5e8163e338bc5f90e9ea2812d50f4f"}
Oct 31 14:15:39.971657 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.971356    4428 scope.go:122] "RemoveContainer" containerID="1e9949005068167b37a79bde2324f9a7546addb54a398e45484b922a74e70472"
Oct 31 14:15:39.972130 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:15:39.972104    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e9949005068167b37a79bde2324f9a7546addb54a398e45484b922a74e70472\": container with ID starting with 1e9949005068167b37a79bde2324f9a7546addb54a398e45484b922a74e70472 not found: ID does not exist" containerID="1e9949005068167b37a79bde2324f9a7546addb54a398e45484b922a74e70472"
Oct 31 14:15:39.972236 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.972141    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e9949005068167b37a79bde2324f9a7546addb54a398e45484b922a74e70472"} err="failed to get container status \"1e9949005068167b37a79bde2324f9a7546addb54a398e45484b922a74e70472\": rpc error: code = NotFound desc = could not find container \"1e9949005068167b37a79bde2324f9a7546addb54a398e45484b922a74e70472\": container with ID starting with 1e9949005068167b37a79bde2324f9a7546addb54a398e45484b922a74e70472 not found: ID does not exist"
Oct 31 14:15:39.972236 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.972167    4428 scope.go:122] "RemoveContainer" containerID="e15c8732e20f1bfffcb329b1c104f1692c77be6852cab8fdc45d2b266f16228c"
Oct 31 14:15:39.986045 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.986027    4428 scope.go:122] "RemoveContainer" containerID="e15c8732e20f1bfffcb329b1c104f1692c77be6852cab8fdc45d2b266f16228c"
Oct 31 14:15:39.986369 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:15:39.986349    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e15c8732e20f1bfffcb329b1c104f1692c77be6852cab8fdc45d2b266f16228c\": container with ID starting with e15c8732e20f1bfffcb329b1c104f1692c77be6852cab8fdc45d2b266f16228c not found: ID does not exist" containerID="e15c8732e20f1bfffcb329b1c104f1692c77be6852cab8fdc45d2b266f16228c"
Oct 31 14:15:39.986429 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.986377    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15c8732e20f1bfffcb329b1c104f1692c77be6852cab8fdc45d2b266f16228c"} err="failed to get container status \"e15c8732e20f1bfffcb329b1c104f1692c77be6852cab8fdc45d2b266f16228c\": rpc error: code = NotFound desc = could not find container \"e15c8732e20f1bfffcb329b1c104f1692c77be6852cab8fdc45d2b266f16228c\": container with ID starting with e15c8732e20f1bfffcb329b1c104f1692c77be6852cab8fdc45d2b266f16228c not found: ID does not exist"
Oct 31 14:15:39.997484 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:39.997457    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"]
Oct 31 14:15:40.005096 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:40.005068    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a9dd7f-86f567bfdgcd6s"]
Oct 31 14:15:40.017694 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:40.017669    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"]
Oct 31 14:15:40.029654 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:40.029614    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dd7f-587bfqsvhf"]
Oct 31 14:15:40.754407 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:40.754371    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21912d67-a608-4417-9191-df1dcc8d958c" path="/var/lib/kubelet/pods/21912d67-a608-4417-9191-df1dcc8d958c/volumes"
Oct 31 14:15:40.754824 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:40.754772    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe" path="/var/lib/kubelet/pods/3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe/volumes"
Oct 31 14:15:44.286156 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.286122    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"]
Oct 31 14:15:44.286724 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.286701    4428 cpu_manager.go:414] "containerMap: removing container" podUID="3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe" containerName="payload-processing"
Oct 31 14:15:44.286821 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.286726    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe" containerName="payload-processing"
Oct 31 14:15:44.286821 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.286743    4428 cpu_manager.go:414] "containerMap: removing container" podUID="43a58059-e4ff-43d0-a312-2cb54fed07c3" containerName="istio-proxy"
Oct 31 14:15:44.286821 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.286751    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a58059-e4ff-43d0-a312-2cb54fed07c3" containerName="istio-proxy"
Oct 31 14:15:44.286821 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.286777    4428 cpu_manager.go:414] "containerMap: removing container" podUID="21912d67-a608-4417-9191-df1dcc8d958c" containerName="payload-pre-processing"
Oct 31 14:15:44.286821 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.286785    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21912d67-a608-4417-9191-df1dcc8d958c" containerName="payload-pre-processing"
Oct 31 14:15:44.286821 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.286795    4428 cpu_manager.go:414] "containerMap: removing container" podUID="43a58059-e4ff-43d0-a312-2cb54fed07c3" containerName="istio-proxy"
Oct 31 14:15:44.286821 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.286802    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a58059-e4ff-43d0-a312-2cb54fed07c3" containerName="istio-proxy"
Oct 31 14:15:44.287213 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.286940    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e2dc027-89b9-4cf9-b96d-bfb87c4d01fe" containerName="payload-processing"
Oct 31 14:15:44.287213 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.286952    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1649b61e-4314-4d04-b696-eebac5aaa77b" containerName="cleanup"
Oct 31 14:15:44.287213 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.286962    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a58059-e4ff-43d0-a312-2cb54fed07c3" containerName="istio-proxy"
Oct 31 14:15:44.287213 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.286976    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="21912d67-a608-4417-9191-df1dcc8d958c" containerName="payload-pre-processing"
Oct 31 14:15:44.287213 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.286986    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a58059-e4ff-43d0-a312-2cb54fed07c3" containerName="istio-proxy"
Oct 31 14:15:44.291934 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.291911    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.299371 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.299328    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-9e257301-kserve-self-signed-certs\""
Oct 31 14:15:44.301122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.301090    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"]
Oct 31 14:15:44.365711 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.365674    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-home\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.365711 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.365710    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfjcv\" (UniqueName: \"kubernetes.io/projected/4a84ab72-3083-4ab2-9b16-792bd60abbe9-kube-api-access-tfjcv\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.365969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.365738    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-model-cache\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.365969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.365909    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-tmp-dir\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.365969 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.365936    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a84ab72-3083-4ab2-9b16-792bd60abbe9-tls-certs\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.366098 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.366075    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-dshm\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.406848 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.406819    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:44.467370 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.467337    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-home\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.467370 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.467371    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfjcv\" (UniqueName: \"kubernetes.io/projected/4a84ab72-3083-4ab2-9b16-792bd60abbe9-kube-api-access-tfjcv\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.467670 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.467395    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-model-cache\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.467670 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.467440    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-tmp-dir\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.467670 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.467462    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a84ab72-3083-4ab2-9b16-792bd60abbe9-tls-certs\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.467670 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.467562    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-dshm\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.467932 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.467814    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-home\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.467932 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.467837    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-model-cache\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.468074 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.468059    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-tmp-dir\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.470133 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.470108    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-dshm\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.470348 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.470328    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a84ab72-3083-4ab2-9b16-792bd60abbe9-tls-certs\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.481446 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.481414    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfjcv\" (UniqueName: \"kubernetes.io/projected/4a84ab72-3083-4ab2-9b16-792bd60abbe9-kube-api-access-tfjcv\") pod \"e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") " pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.607787 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.607684    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:44.890479 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.890448    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"]
Oct 31 14:15:44.893147 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:15:44.893108    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a84ab72_3083_4ab2_9b16_792bd60abbe9.slice/crio-5ad654178b233e8efedc93f7b4fcc7044fa114cf5ffaea4e4f26c4c93d1b8e09 WatchSource:0}: Error finding container 5ad654178b233e8efedc93f7b4fcc7044fa114cf5ffaea4e4f26c4c93d1b8e09: Status 404 returned error can't find the container with id 5ad654178b233e8efedc93f7b4fcc7044fa114cf5ffaea4e4f26c4c93d1b8e09
Oct 31 14:15:44.993905 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.993844    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq" event={"ID":"4a84ab72-3083-4ab2-9b16-792bd60abbe9","Type":"ContainerStarted","Data":"c83756cc2fc4e4cb76ff84e1e584e2981fdb83724d40542c364dad19062f7dc6"}
Oct 31 14:15:44.994095 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:44.993916    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq" event={"ID":"4a84ab72-3083-4ab2-9b16-792bd60abbe9","Type":"ContainerStarted","Data":"5ad654178b233e8efedc93f7b4fcc7044fa114cf5ffaea4e4f26c4c93d1b8e09"}
Oct 31 14:15:45.014794 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:45.014739    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq" podStartSLOduration=1.014723108 podStartE2EDuration="1.014723108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:15:45.01290128 +0000 UTC m=+4342.411249904" watchObservedRunningTime="2026-10-31 14:15:45.014723108 +0000 UTC m=+4342.413071731"
Oct 31 14:15:45.275841 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:45.275800    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:45.608388 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:45.608310    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:47.933378 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:47.933339    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:48.899895 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:15:48.899463    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:15:54.122366 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:54.122327    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:15:54.608530 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:54.608492    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:54.621019 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:54.620992    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:15:55.133997 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:15:55.133966    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:16:00.584624 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:00.584588    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"]
Oct 31 14:16:00.588398 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:00.588374    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"
Oct 31 14:16:00.597002 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:00.596233    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:16:00.597002 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:00.596343    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:16:00.598347 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:00.598323    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"]
Oct 31 14:16:00.602824 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:00.602798    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kznj\" (UniqueName: \"kubernetes.io/projected/20d5d735-f0a7-4756-96c1-7fbfa8018803-kube-api-access-9kznj\") pod \"maas-api-revoke-sub-452b6d7b-kqmz2\" (UID: \"20d5d735-f0a7-4756-96c1-7fbfa8018803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"
Oct 31 14:16:00.602984 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:00.602842    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d5d735-f0a7-4756-96c1-7fbfa8018803-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-kqmz2\" (UID: \"20d5d735-f0a7-4756-96c1-7fbfa8018803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"
Oct 31 14:16:00.602984 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:00.602916    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d5d735-f0a7-4756-96c1-7fbfa8018803-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-kqmz2\" (UID: \"20d5d735-f0a7-4756-96c1-7fbfa8018803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"
Oct 31 14:16:00.703963 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:00.703929    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kznj\" (UniqueName: \"kubernetes.io/projected/20d5d735-f0a7-4756-96c1-7fbfa8018803-kube-api-access-9kznj\") pod \"maas-api-revoke-sub-452b6d7b-kqmz2\" (UID: \"20d5d735-f0a7-4756-96c1-7fbfa8018803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"
Oct 31 14:16:00.703963 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:00.703971    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d5d735-f0a7-4756-96c1-7fbfa8018803-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-kqmz2\" (UID: \"20d5d735-f0a7-4756-96c1-7fbfa8018803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"
Oct 31 14:16:00.704223 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:00.704017    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d5d735-f0a7-4756-96c1-7fbfa8018803-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-kqmz2\" (UID: \"20d5d735-f0a7-4756-96c1-7fbfa8018803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"
Oct 31 14:16:00.704680 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:00.704657    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d5d735-f0a7-4756-96c1-7fbfa8018803-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-kqmz2\" (UID: \"20d5d735-f0a7-4756-96c1-7fbfa8018803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"
Oct 31 14:16:00.706723 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:00.706701    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d5d735-f0a7-4756-96c1-7fbfa8018803-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-kqmz2\" (UID: \"20d5d735-f0a7-4756-96c1-7fbfa8018803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"
Oct 31 14:16:00.717425 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:00.717392    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kznj\" (UniqueName: \"kubernetes.io/projected/20d5d735-f0a7-4756-96c1-7fbfa8018803-kube-api-access-9kznj\") pod \"maas-api-revoke-sub-452b6d7b-kqmz2\" (UID: \"20d5d735-f0a7-4756-96c1-7fbfa8018803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"
Oct 31 14:16:00.905973 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:00.905845    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"
Oct 31 14:16:01.191616 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:01.191561    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"]
Oct 31 14:16:01.196701 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:16:01.196654    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d5d735_f0a7_4756_96c1_7fbfa8018803.slice/crio-b5f6d8570a8b215ee2f8482bf570ac350cc3b99069ce660f11f32d95feebdb27 WatchSource:0}: Error finding container b5f6d8570a8b215ee2f8482bf570ac350cc3b99069ce660f11f32d95feebdb27: Status 404 returned error can't find the container with id b5f6d8570a8b215ee2f8482bf570ac350cc3b99069ce660f11f32d95feebdb27
Oct 31 14:16:02.164117 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:02.164071    4428 generic.go:356] "Generic (PLEG): container finished" podID="20d5d735-f0a7-4756-96c1-7fbfa8018803" containerID="e70e63e4fad4ab17e6ee49aea4262509cfb1a712fa7ba556cb1bd4d6d1b7a8b8" exitCode=0
Oct 31 14:16:02.164546 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:02.164155    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2" event={"ID":"20d5d735-f0a7-4756-96c1-7fbfa8018803","Type":"ContainerDied","Data":"e70e63e4fad4ab17e6ee49aea4262509cfb1a712fa7ba556cb1bd4d6d1b7a8b8"}
Oct 31 14:16:02.164546 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:02.164221    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2" event={"ID":"20d5d735-f0a7-4756-96c1-7fbfa8018803","Type":"ContainerStarted","Data":"b5f6d8570a8b215ee2f8482bf570ac350cc3b99069ce660f11f32d95feebdb27"}
Oct 31 14:16:03.803896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:03.792102    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"
Oct 31 14:16:03.817896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:03.809514    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20d5d735-f0a7-4756-96c1-7fbfa8018803-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d5d735-f0a7-4756-96c1-7fbfa8018803-maas-api-xks-ca\") pod \"20d5d735-f0a7-4756-96c1-7fbfa8018803\" (UID: \"20d5d735-f0a7-4756-96c1-7fbfa8018803\") "
Oct 31 14:16:03.841097 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:03.837368    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d5d735-f0a7-4756-96c1-7fbfa8018803-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d5d735-f0a7-4756-96c1-7fbfa8018803-maas-api-openshift-ca\") pod \"20d5d735-f0a7-4756-96c1-7fbfa8018803\" (UID: \"20d5d735-f0a7-4756-96c1-7fbfa8018803\") "
Oct 31 14:16:03.841097 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:03.837444    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d5d735-f0a7-4756-96c1-7fbfa8018803-kube-api-access-9kznj\" (UniqueName: \"kubernetes.io/projected/20d5d735-f0a7-4756-96c1-7fbfa8018803-kube-api-access-9kznj\") pod \"20d5d735-f0a7-4756-96c1-7fbfa8018803\" (UID: \"20d5d735-f0a7-4756-96c1-7fbfa8018803\") "
Oct 31 14:16:03.841097 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:03.832744    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d5d735-f0a7-4756-96c1-7fbfa8018803-maas-api-xks-ca" pod "20d5d735-f0a7-4756-96c1-7fbfa8018803" (UID: "20d5d735-f0a7-4756-96c1-7fbfa8018803"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:16:03.841097 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:03.839024    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d5d735-f0a7-4756-96c1-7fbfa8018803-maas-api-openshift-ca" pod "20d5d735-f0a7-4756-96c1-7fbfa8018803" (UID: "20d5d735-f0a7-4756-96c1-7fbfa8018803"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:16:03.841678 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:03.841644    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d5d735-f0a7-4756-96c1-7fbfa8018803-kube-api-access-9kznj" pod "20d5d735-f0a7-4756-96c1-7fbfa8018803" (UID: "20d5d735-f0a7-4756-96c1-7fbfa8018803"). InnerVolumeSpecName "kube-api-access-9kznj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:16:03.939759 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:03.939595    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d5d735-f0a7-4756-96c1-7fbfa8018803-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:03.939759 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:03.939643    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d5d735-f0a7-4756-96c1-7fbfa8018803-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:03.939759 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:03.939665    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kznj\" (UniqueName: \"kubernetes.io/projected/20d5d735-f0a7-4756-96c1-7fbfa8018803-kube-api-access-9kznj\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:04.176240 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:04.175945    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:16:04.230755 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:04.229979    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"
Oct 31 14:16:04.230755 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:04.230024    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2" event={"ID":"20d5d735-f0a7-4756-96c1-7fbfa8018803","Type":"ContainerDied","Data":"b5f6d8570a8b215ee2f8482bf570ac350cc3b99069ce660f11f32d95feebdb27"}
Oct 31 14:16:04.230755 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:04.230095    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5f6d8570a8b215ee2f8482bf570ac350cc3b99069ce660f11f32d95feebdb27"
Oct 31 14:16:04.337201 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:04.337159    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:16:11.664642 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:11.664604    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"]
Oct 31 14:16:11.768496 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:11.768459    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:16:13.137569 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:13.137526    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"]
Oct 31 14:16:13.272661 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:13.272625    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:16:16.580464 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:16.580426    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:16:16.741245 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:16.741208    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"]
Oct 31 14:16:16.741655 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:16.741640    4428 cpu_manager.go:414] "containerMap: removing container" podUID="20d5d735-f0a7-4756-96c1-7fbfa8018803" containerName="revoke-keys"
Oct 31 14:16:16.741712 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:16.741657    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d5d735-f0a7-4756-96c1-7fbfa8018803" containerName="revoke-keys"
Oct 31 14:16:16.741751 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:16.741736    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d5d735-f0a7-4756-96c1-7fbfa8018803" containerName="revoke-keys"
Oct 31 14:16:16.745172 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:16.745153    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"
Oct 31 14:16:16.753235 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:16.753202    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:16:16.754710 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:16.754309    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:16:16.760826 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:16.760798    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"]
Oct 31 14:16:16.926449 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:16.926335    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64524ae5-3f14-4ed5-9b9a-a3187b168224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-db7l6\" (UID: \"64524ae5-3f14-4ed5-9b9a-a3187b168224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"
Oct 31 14:16:16.926659 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:16.926574    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64524ae5-3f14-4ed5-9b9a-a3187b168224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-db7l6\" (UID: \"64524ae5-3f14-4ed5-9b9a-a3187b168224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"
Oct 31 14:16:16.926659 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:16.926609    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st8xb\" (UniqueName: \"kubernetes.io/projected/64524ae5-3f14-4ed5-9b9a-a3187b168224-kube-api-access-st8xb\") pod \"maas-api-revoke-sub-3e31a506-db7l6\" (UID: \"64524ae5-3f14-4ed5-9b9a-a3187b168224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"
Oct 31 14:16:17.028144 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:17.028106    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64524ae5-3f14-4ed5-9b9a-a3187b168224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-db7l6\" (UID: \"64524ae5-3f14-4ed5-9b9a-a3187b168224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"
Oct 31 14:16:17.028383 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:17.028214    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64524ae5-3f14-4ed5-9b9a-a3187b168224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-db7l6\" (UID: \"64524ae5-3f14-4ed5-9b9a-a3187b168224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"
Oct 31 14:16:17.028383 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:17.028234    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st8xb\" (UniqueName: \"kubernetes.io/projected/64524ae5-3f14-4ed5-9b9a-a3187b168224-kube-api-access-st8xb\") pod \"maas-api-revoke-sub-3e31a506-db7l6\" (UID: \"64524ae5-3f14-4ed5-9b9a-a3187b168224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"
Oct 31 14:16:17.028889 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:17.028843    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64524ae5-3f14-4ed5-9b9a-a3187b168224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-db7l6\" (UID: \"64524ae5-3f14-4ed5-9b9a-a3187b168224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"
Oct 31 14:16:17.030849 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:17.030828    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64524ae5-3f14-4ed5-9b9a-a3187b168224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-db7l6\" (UID: \"64524ae5-3f14-4ed5-9b9a-a3187b168224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"
Oct 31 14:16:17.041694 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:17.041670    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st8xb\" (UniqueName: \"kubernetes.io/projected/64524ae5-3f14-4ed5-9b9a-a3187b168224-kube-api-access-st8xb\") pod \"maas-api-revoke-sub-3e31a506-db7l6\" (UID: \"64524ae5-3f14-4ed5-9b9a-a3187b168224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"
Oct 31 14:16:17.068680 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:17.068653    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"
Oct 31 14:16:17.301388 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:17.301311    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:16:17.360558 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:17.360523    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"]
Oct 31 14:16:17.363027 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:16:17.362991    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64524ae5_3f14_4ed5_9b9a_a3187b168224.slice/crio-f01e426693edbd1e6a08ffc19d8d47bf3512c9e5d25b7353653e41a5986b0014 WatchSource:0}: Error finding container f01e426693edbd1e6a08ffc19d8d47bf3512c9e5d25b7353653e41a5986b0014: Status 404 returned error can't find the container with id f01e426693edbd1e6a08ffc19d8d47bf3512c9e5d25b7353653e41a5986b0014
Oct 31 14:16:18.341943 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:18.341907    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6" event={"ID":"64524ae5-3f14-4ed5-9b9a-a3187b168224","Type":"ContainerStarted","Data":"d2d9071be184af98632640cd54e19def3726d4d599d625ecc9caf63c4a49b13b"}
Oct 31 14:16:18.342410 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:18.341987    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6" event={"ID":"64524ae5-3f14-4ed5-9b9a-a3187b168224","Type":"ContainerStarted","Data":"f01e426693edbd1e6a08ffc19d8d47bf3512c9e5d25b7353653e41a5986b0014"}
Oct 31 14:16:18.396900 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:18.396111    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6" podStartSLOduration=2.396083485 podStartE2EDuration="2.396083485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:16:18.37528062 +0000 UTC m=+4375.773629256" watchObservedRunningTime="2026-10-31 14:16:18.396083485 +0000 UTC m=+4375.794432109"
Oct 31 14:16:18.831901 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:16:18.829467    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:16:19.357896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:19.356360    4428 generic.go:356] "Generic (PLEG): container finished" podID="64524ae5-3f14-4ed5-9b9a-a3187b168224" containerID="d2d9071be184af98632640cd54e19def3726d4d599d625ecc9caf63c4a49b13b" exitCode=0
Oct 31 14:16:19.357896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:19.356511    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6" event={"ID":"64524ae5-3f14-4ed5-9b9a-a3187b168224","Type":"ContainerDied","Data":"d2d9071be184af98632640cd54e19def3726d4d599d625ecc9caf63c4a49b13b"}
Oct 31 14:16:20.503481 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:20.503452    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"
Oct 31 14:16:20.644433 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:20.644343    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64524ae5-3f14-4ed5-9b9a-a3187b168224-kube-api-access-st8xb\" (UniqueName: \"kubernetes.io/projected/64524ae5-3f14-4ed5-9b9a-a3187b168224-kube-api-access-st8xb\") pod \"64524ae5-3f14-4ed5-9b9a-a3187b168224\" (UID: \"64524ae5-3f14-4ed5-9b9a-a3187b168224\") "
Oct 31 14:16:20.644617 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:20.644454    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64524ae5-3f14-4ed5-9b9a-a3187b168224-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64524ae5-3f14-4ed5-9b9a-a3187b168224-maas-api-xks-ca\") pod \"64524ae5-3f14-4ed5-9b9a-a3187b168224\" (UID: \"64524ae5-3f14-4ed5-9b9a-a3187b168224\") "
Oct 31 14:16:20.644617 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:20.644578    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64524ae5-3f14-4ed5-9b9a-a3187b168224-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64524ae5-3f14-4ed5-9b9a-a3187b168224-maas-api-openshift-ca\") pod \"64524ae5-3f14-4ed5-9b9a-a3187b168224\" (UID: \"64524ae5-3f14-4ed5-9b9a-a3187b168224\") "
Oct 31 14:16:20.644978 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:20.644953    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64524ae5-3f14-4ed5-9b9a-a3187b168224-maas-api-openshift-ca" pod "64524ae5-3f14-4ed5-9b9a-a3187b168224" (UID: "64524ae5-3f14-4ed5-9b9a-a3187b168224"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:16:20.646746 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:20.646714    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64524ae5-3f14-4ed5-9b9a-a3187b168224-kube-api-access-st8xb" pod "64524ae5-3f14-4ed5-9b9a-a3187b168224" (UID: "64524ae5-3f14-4ed5-9b9a-a3187b168224"). InnerVolumeSpecName "kube-api-access-st8xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:16:20.647048 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:20.647024    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64524ae5-3f14-4ed5-9b9a-a3187b168224-maas-api-xks-ca" pod "64524ae5-3f14-4ed5-9b9a-a3187b168224" (UID: "64524ae5-3f14-4ed5-9b9a-a3187b168224"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:16:20.746183 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:20.746140    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64524ae5-3f14-4ed5-9b9a-a3187b168224-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:20.746183 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:20.746173    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st8xb\" (UniqueName: \"kubernetes.io/projected/64524ae5-3f14-4ed5-9b9a-a3187b168224-kube-api-access-st8xb\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:20.746183 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:20.746184    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64524ae5-3f14-4ed5-9b9a-a3187b168224-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:21.372060 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:21.372014    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6" event={"ID":"64524ae5-3f14-4ed5-9b9a-a3187b168224","Type":"ContainerDied","Data":"f01e426693edbd1e6a08ffc19d8d47bf3512c9e5d25b7353653e41a5986b0014"}
Oct 31 14:16:21.372313 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:21.372102    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f01e426693edbd1e6a08ffc19d8d47bf3512c9e5d25b7353653e41a5986b0014"
Oct 31 14:16:21.372313 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:21.372175    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"
Oct 31 14:16:26.695748 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:26.695682    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt" podUID="5f404e76-c160-441d-8ac3-0a0a36fba326" containerName="main" containerID="cri-o://a631a7dd0a0cc646f9c50aec8d44b9372acda6f8aa40db94d6383043134cce51" gracePeriod=45
Oct 31 14:16:26.959737 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:26.959664    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:16:27.007236 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.007198    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-dshm\") pod \"5f404e76-c160-441d-8ac3-0a0a36fba326\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") "
Oct 31 14:16:27.007479 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.007303    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f404e76-c160-441d-8ac3-0a0a36fba326-kube-api-access-98lrt\" (UniqueName: \"kubernetes.io/projected/5f404e76-c160-441d-8ac3-0a0a36fba326-kube-api-access-98lrt\") pod \"5f404e76-c160-441d-8ac3-0a0a36fba326\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") "
Oct 31 14:16:27.007479 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.007362    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-home\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-home\") pod \"5f404e76-c160-441d-8ac3-0a0a36fba326\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") "
Oct 31 14:16:27.007479 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.007393    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f404e76-c160-441d-8ac3-0a0a36fba326-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f404e76-c160-441d-8ac3-0a0a36fba326-tls-certs\") pod \"5f404e76-c160-441d-8ac3-0a0a36fba326\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") "
Oct 31 14:16:27.007479 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.007423    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-model-cache\") pod \"5f404e76-c160-441d-8ac3-0a0a36fba326\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") "
Oct 31 14:16:27.007479 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.007470    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-tmp-dir\") pod \"5f404e76-c160-441d-8ac3-0a0a36fba326\" (UID: \"5f404e76-c160-441d-8ac3-0a0a36fba326\") "
Oct 31 14:16:27.007782 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.007623    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-home" pod "5f404e76-c160-441d-8ac3-0a0a36fba326" (UID: "5f404e76-c160-441d-8ac3-0a0a36fba326"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:16:27.007898 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.007846    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:27.007975 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.007896    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-tmp-dir" pod "5f404e76-c160-441d-8ac3-0a0a36fba326" (UID: "5f404e76-c160-441d-8ac3-0a0a36fba326"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:16:27.008129 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.008106    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-model-cache" pod "5f404e76-c160-441d-8ac3-0a0a36fba326" (UID: "5f404e76-c160-441d-8ac3-0a0a36fba326"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:16:27.010133 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.010108    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f404e76-c160-441d-8ac3-0a0a36fba326-tls-certs" pod "5f404e76-c160-441d-8ac3-0a0a36fba326" (UID: "5f404e76-c160-441d-8ac3-0a0a36fba326"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:16:27.010290 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.010269    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-dshm" pod "5f404e76-c160-441d-8ac3-0a0a36fba326" (UID: "5f404e76-c160-441d-8ac3-0a0a36fba326"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:16:27.010340 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.010300    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f404e76-c160-441d-8ac3-0a0a36fba326-kube-api-access-98lrt" pod "5f404e76-c160-441d-8ac3-0a0a36fba326" (UID: "5f404e76-c160-441d-8ac3-0a0a36fba326"). InnerVolumeSpecName "kube-api-access-98lrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:16:27.109386 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.109342    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:27.109386 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.109379    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98lrt\" (UniqueName: \"kubernetes.io/projected/5f404e76-c160-441d-8ac3-0a0a36fba326-kube-api-access-98lrt\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:27.109386 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.109389    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f404e76-c160-441d-8ac3-0a0a36fba326-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:27.109747 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.109399    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:27.109747 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.109408    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f404e76-c160-441d-8ac3-0a0a36fba326-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:27.404577 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.404539    4428 generic.go:356] "Generic (PLEG): container finished" podID="5f404e76-c160-441d-8ac3-0a0a36fba326" containerID="a631a7dd0a0cc646f9c50aec8d44b9372acda6f8aa40db94d6383043134cce51" exitCode=0
Oct 31 14:16:27.404772 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.404582    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"
Oct 31 14:16:27.404772 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.404621    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt" event={"ID":"5f404e76-c160-441d-8ac3-0a0a36fba326","Type":"ContainerDied","Data":"a631a7dd0a0cc646f9c50aec8d44b9372acda6f8aa40db94d6383043134cce51"}
Oct 31 14:16:27.404772 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.404670    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt" event={"ID":"5f404e76-c160-441d-8ac3-0a0a36fba326","Type":"ContainerDied","Data":"31d5d37ff96aea8507a6c6580cfb520f597ecdbd5e8d2ca102c11fb0b5c5f2d9"}
Oct 31 14:16:27.404772 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.404688    4428 scope.go:122] "RemoveContainer" containerID="a631a7dd0a0cc646f9c50aec8d44b9372acda6f8aa40db94d6383043134cce51"
Oct 31 14:16:27.414431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.414410    4428 scope.go:122] "RemoveContainer" containerID="a631a7dd0a0cc646f9c50aec8d44b9372acda6f8aa40db94d6383043134cce51"
Oct 31 14:16:27.414711 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:16:27.414691    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a631a7dd0a0cc646f9c50aec8d44b9372acda6f8aa40db94d6383043134cce51\": container with ID starting with a631a7dd0a0cc646f9c50aec8d44b9372acda6f8aa40db94d6383043134cce51 not found: ID does not exist" containerID="a631a7dd0a0cc646f9c50aec8d44b9372acda6f8aa40db94d6383043134cce51"
Oct 31 14:16:27.414766 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.414721    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a631a7dd0a0cc646f9c50aec8d44b9372acda6f8aa40db94d6383043134cce51"} err="failed to get container status \"a631a7dd0a0cc646f9c50aec8d44b9372acda6f8aa40db94d6383043134cce51\": rpc error: code = NotFound desc = could not find container \"a631a7dd0a0cc646f9c50aec8d44b9372acda6f8aa40db94d6383043134cce51\": container with ID starting with a631a7dd0a0cc646f9c50aec8d44b9372acda6f8aa40db94d6383043134cce51 not found: ID does not exist"
Oct 31 14:16:27.427241 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.427217    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"]
Oct 31 14:16:27.433649 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.433621    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt"]
Oct 31 14:16:27.955424 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:27.955369    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-9e257301-kserve-7757f88cc-cc2wt" podUID="5f404e76-c160-441d-8ac3-0a0a36fba326" containerName="main" probeResult="failure" output="Get \"https://10.134.1.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 14:16:28.161375 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:28.161327    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq" podUID="4a84ab72-3083-4ab2-9b16-792bd60abbe9" containerName="main" containerID="cri-o://c83756cc2fc4e4cb76ff84e1e584e2981fdb83724d40542c364dad19062f7dc6" gracePeriod=45
Oct 31 14:16:28.430909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:28.428935    4428 generic.go:356] "Generic (PLEG): container finished" podID="4a84ab72-3083-4ab2-9b16-792bd60abbe9" containerID="c83756cc2fc4e4cb76ff84e1e584e2981fdb83724d40542c364dad19062f7dc6" exitCode=0
Oct 31 14:16:28.430909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:28.429111    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq" event={"ID":"4a84ab72-3083-4ab2-9b16-792bd60abbe9","Type":"ContainerDied","Data":"c83756cc2fc4e4cb76ff84e1e584e2981fdb83724d40542c364dad19062f7dc6"}
Oct 31 14:16:28.818940 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:28.818784    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f404e76-c160-441d-8ac3-0a0a36fba326" path="/var/lib/kubelet/pods/5f404e76-c160-441d-8ac3-0a0a36fba326/volumes"
Oct 31 14:16:28.833951 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:28.826053    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:16:28.941434 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:28.941367    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-model-cache\") pod \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") "
Oct 31 14:16:28.941675 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:28.941451    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-dshm\") pod \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") "
Oct 31 14:16:28.941675 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:28.941513    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a84ab72-3083-4ab2-9b16-792bd60abbe9-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a84ab72-3083-4ab2-9b16-792bd60abbe9-tls-certs\") pod \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") "
Oct 31 14:16:28.941675 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:28.941546    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-home\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-home\") pod \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") "
Oct 31 14:16:28.941675 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:28.941592    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-tmp-dir\") pod \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") "
Oct 31 14:16:28.941675 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:28.941637    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a84ab72-3083-4ab2-9b16-792bd60abbe9-kube-api-access-tfjcv\" (UniqueName: \"kubernetes.io/projected/4a84ab72-3083-4ab2-9b16-792bd60abbe9-kube-api-access-tfjcv\") pod \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\" (UID: \"4a84ab72-3083-4ab2-9b16-792bd60abbe9\") "
Oct 31 14:16:28.942801 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:28.942770    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-model-cache" pod "4a84ab72-3083-4ab2-9b16-792bd60abbe9" (UID: "4a84ab72-3083-4ab2-9b16-792bd60abbe9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:16:28.942989 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:28.942966    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-tmp-dir" pod "4a84ab72-3083-4ab2-9b16-792bd60abbe9" (UID: "4a84ab72-3083-4ab2-9b16-792bd60abbe9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:16:28.943081 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:28.943008    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-home" pod "4a84ab72-3083-4ab2-9b16-792bd60abbe9" (UID: "4a84ab72-3083-4ab2-9b16-792bd60abbe9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:16:28.944799 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:28.944765    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a84ab72-3083-4ab2-9b16-792bd60abbe9-kube-api-access-tfjcv" pod "4a84ab72-3083-4ab2-9b16-792bd60abbe9" (UID: "4a84ab72-3083-4ab2-9b16-792bd60abbe9"). InnerVolumeSpecName "kube-api-access-tfjcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:16:28.979914 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:28.969018    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-dshm" pod "4a84ab72-3083-4ab2-9b16-792bd60abbe9" (UID: "4a84ab72-3083-4ab2-9b16-792bd60abbe9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:16:29.032950 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:29.032850    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a84ab72-3083-4ab2-9b16-792bd60abbe9-tls-certs" pod "4a84ab72-3083-4ab2-9b16-792bd60abbe9" (UID: "4a84ab72-3083-4ab2-9b16-792bd60abbe9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:16:29.043466 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:29.043425    4428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-model-cache\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:29.043466 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:29.043461    4428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-dshm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:29.043682 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:29.043485    4428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a84ab72-3083-4ab2-9b16-792bd60abbe9-tls-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:29.043682 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:29.043499    4428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-home\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:29.043682 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:29.043511    4428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a84ab72-3083-4ab2-9b16-792bd60abbe9-tmp-dir\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:29.043682 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:29.043521    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfjcv\" (UniqueName: \"kubernetes.io/projected/4a84ab72-3083-4ab2-9b16-792bd60abbe9-kube-api-access-tfjcv\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:29.439651 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:29.439599    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq" event={"ID":"4a84ab72-3083-4ab2-9b16-792bd60abbe9","Type":"ContainerDied","Data":"5ad654178b233e8efedc93f7b4fcc7044fa114cf5ffaea4e4f26c4c93d1b8e09"}
Oct 31 14:16:29.439651 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:29.439658    4428 scope.go:122] "RemoveContainer" containerID="c83756cc2fc4e4cb76ff84e1e584e2981fdb83724d40542c364dad19062f7dc6"
Oct 31 14:16:29.440001 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:29.439789    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"
Oct 31 14:16:29.508765 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:29.508650    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"]
Oct 31 14:16:29.509108 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:29.509085    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-9e257301-kserve-5c7c8c6f89-7kdcq"]
Oct 31 14:16:30.757757 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:30.757708    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a84ab72-3083-4ab2-9b16-792bd60abbe9" path="/var/lib/kubelet/pods/4a84ab72-3083-4ab2-9b16-792bd60abbe9/volumes"
Oct 31 14:16:32.114923 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:32.114726    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-n56bm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 14:16:32.114923 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:32.114840    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-n56bm" podUID="aaa34a48-48ac-4d05-8d9d-0d0c0a870f88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 14:16:36.490541 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:36.490481    4428 scope.go:122] "RemoveContainer" containerID="39c1dade2bcb2df30e8839580c09d36eb51794144cf6d71075cdee8477178106"
Oct 31 14:16:36.500565 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:36.500544    4428 scope.go:122] "RemoveContainer" containerID="f499b613fcebbf6c206ee42a13b9def73660f9f296d68d3338ada5939688f3ce"
Oct 31 14:16:38.350624 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.350575    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"]
Oct 31 14:16:38.351251 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.351223    4428 cpu_manager.go:414] "containerMap: removing container" podUID="5f404e76-c160-441d-8ac3-0a0a36fba326" containerName="main"
Oct 31 14:16:38.351251 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.351252    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f404e76-c160-441d-8ac3-0a0a36fba326" containerName="main"
Oct 31 14:16:38.351452 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.351289    4428 cpu_manager.go:414] "containerMap: removing container" podUID="64524ae5-3f14-4ed5-9b9a-a3187b168224" containerName="revoke-keys"
Oct 31 14:16:38.351452 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.351298    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64524ae5-3f14-4ed5-9b9a-a3187b168224" containerName="revoke-keys"
Oct 31 14:16:38.351452 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.351313    4428 cpu_manager.go:414] "containerMap: removing container" podUID="4a84ab72-3083-4ab2-9b16-792bd60abbe9" containerName="main"
Oct 31 14:16:38.351452 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.351321    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a84ab72-3083-4ab2-9b16-792bd60abbe9" containerName="main"
Oct 31 14:16:38.351452 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.351423    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="64524ae5-3f14-4ed5-9b9a-a3187b168224" containerName="revoke-keys"
Oct 31 14:16:38.351452 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.351431    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a84ab72-3083-4ab2-9b16-792bd60abbe9" containerName="main"
Oct 31 14:16:38.351452 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.351442    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f404e76-c160-441d-8ac3-0a0a36fba326" containerName="main"
Oct 31 14:16:38.356081 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.356061    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"
Oct 31 14:16:38.362362 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.362323    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:16:38.362546 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.362528    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\""
Oct 31 14:16:38.364563 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.364527    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"]
Oct 31 14:16:38.383647 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.383611    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d500889f-a602-4b38-b66c-db0f5500c483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ec5dc37-xbjjf\" (UID: \"d500889f-a602-4b38-b66c-db0f5500c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"
Oct 31 14:16:38.383856 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.383709    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d500889f-a602-4b38-b66c-db0f5500c483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ec5dc37-xbjjf\" (UID: \"d500889f-a602-4b38-b66c-db0f5500c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"
Oct 31 14:16:38.383856 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.383778    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7lx6\" (UniqueName: \"kubernetes.io/projected/d500889f-a602-4b38-b66c-db0f5500c483-kube-api-access-z7lx6\") pod \"maas-api-revoke-sub-2ec5dc37-xbjjf\" (UID: \"d500889f-a602-4b38-b66c-db0f5500c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"
Oct 31 14:16:38.485087 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.485044    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d500889f-a602-4b38-b66c-db0f5500c483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ec5dc37-xbjjf\" (UID: \"d500889f-a602-4b38-b66c-db0f5500c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"
Oct 31 14:16:38.485287 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.485120    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d500889f-a602-4b38-b66c-db0f5500c483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ec5dc37-xbjjf\" (UID: \"d500889f-a602-4b38-b66c-db0f5500c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"
Oct 31 14:16:38.485287 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.485195    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7lx6\" (UniqueName: \"kubernetes.io/projected/d500889f-a602-4b38-b66c-db0f5500c483-kube-api-access-z7lx6\") pod \"maas-api-revoke-sub-2ec5dc37-xbjjf\" (UID: \"d500889f-a602-4b38-b66c-db0f5500c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"
Oct 31 14:16:38.485915 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.485884    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d500889f-a602-4b38-b66c-db0f5500c483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ec5dc37-xbjjf\" (UID: \"d500889f-a602-4b38-b66c-db0f5500c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"
Oct 31 14:16:38.487640 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.487616    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d500889f-a602-4b38-b66c-db0f5500c483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ec5dc37-xbjjf\" (UID: \"d500889f-a602-4b38-b66c-db0f5500c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"
Oct 31 14:16:38.498710 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.498683    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7lx6\" (UniqueName: \"kubernetes.io/projected/d500889f-a602-4b38-b66c-db0f5500c483-kube-api-access-z7lx6\") pod \"maas-api-revoke-sub-2ec5dc37-xbjjf\" (UID: \"d500889f-a602-4b38-b66c-db0f5500c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"
Oct 31 14:16:38.672530 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.672441    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"
Oct 31 14:16:38.953972 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:38.953883    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"]
Oct 31 14:16:38.957254 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:16:38.957224    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd500889f_a602_4b38_b66c_db0f5500c483.slice/crio-1f6868f10bdc272d30780952c0657bfc98394da05f592988423faa3f182bc141 WatchSource:0}: Error finding container 1f6868f10bdc272d30780952c0657bfc98394da05f592988423faa3f182bc141: Status 404 returned error can't find the container with id 1f6868f10bdc272d30780952c0657bfc98394da05f592988423faa3f182bc141
Oct 31 14:16:39.516171 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:39.516074    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf" event={"ID":"d500889f-a602-4b38-b66c-db0f5500c483","Type":"ContainerStarted","Data":"359fa65054fc32d37380f4713e69543d61822f52f2e23d7a46f8c65138c4ab10"}
Oct 31 14:16:39.516171 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:39.516132    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf" event={"ID":"d500889f-a602-4b38-b66c-db0f5500c483","Type":"ContainerStarted","Data":"1f6868f10bdc272d30780952c0657bfc98394da05f592988423faa3f182bc141"}
Oct 31 14:16:39.537266 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:39.537212    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf" podStartSLOduration=1.53719949 podStartE2EDuration="1.53719949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:16:39.534799327 +0000 UTC m=+4396.933147949" watchObservedRunningTime="2026-10-31 14:16:39.53719949 +0000 UTC m=+4396.935548134"
Oct 31 14:16:40.523110 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:40.523064    4428 generic.go:356] "Generic (PLEG): container finished" podID="d500889f-a602-4b38-b66c-db0f5500c483" containerID="359fa65054fc32d37380f4713e69543d61822f52f2e23d7a46f8c65138c4ab10" exitCode=0
Oct 31 14:16:40.523617 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:40.523131    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf" event={"ID":"d500889f-a602-4b38-b66c-db0f5500c483","Type":"ContainerDied","Data":"359fa65054fc32d37380f4713e69543d61822f52f2e23d7a46f8c65138c4ab10"}
Oct 31 14:16:41.026668 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:41.026633    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"]
Oct 31 14:16:41.029707 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:41.029675    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgzbt"]
Oct 31 14:16:41.676045 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:41.676015    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"
Oct 31 14:16:41.716909 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:41.716860    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d500889f-a602-4b38-b66c-db0f5500c483-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d500889f-a602-4b38-b66c-db0f5500c483-maas-api-openshift-ca\") pod \"d500889f-a602-4b38-b66c-db0f5500c483\" (UID: \"d500889f-a602-4b38-b66c-db0f5500c483\") "
Oct 31 14:16:41.717129 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:41.716926    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d500889f-a602-4b38-b66c-db0f5500c483-kube-api-access-z7lx6\" (UniqueName: \"kubernetes.io/projected/d500889f-a602-4b38-b66c-db0f5500c483-kube-api-access-z7lx6\") pod \"d500889f-a602-4b38-b66c-db0f5500c483\" (UID: \"d500889f-a602-4b38-b66c-db0f5500c483\") "
Oct 31 14:16:41.717129 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:41.716964    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d500889f-a602-4b38-b66c-db0f5500c483-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d500889f-a602-4b38-b66c-db0f5500c483-maas-api-xks-ca\") pod \"d500889f-a602-4b38-b66c-db0f5500c483\" (UID: \"d500889f-a602-4b38-b66c-db0f5500c483\") "
Oct 31 14:16:41.717277 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:41.717206    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d500889f-a602-4b38-b66c-db0f5500c483-maas-api-openshift-ca" pod "d500889f-a602-4b38-b66c-db0f5500c483" (UID: "d500889f-a602-4b38-b66c-db0f5500c483"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:16:41.719276 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:41.719242    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d500889f-a602-4b38-b66c-db0f5500c483-maas-api-xks-ca" pod "d500889f-a602-4b38-b66c-db0f5500c483" (UID: "d500889f-a602-4b38-b66c-db0f5500c483"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:16:41.719414 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:41.719307    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d500889f-a602-4b38-b66c-db0f5500c483-kube-api-access-z7lx6" pod "d500889f-a602-4b38-b66c-db0f5500c483" (UID: "d500889f-a602-4b38-b66c-db0f5500c483"). InnerVolumeSpecName "kube-api-access-z7lx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:16:41.818995 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:41.818896    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d500889f-a602-4b38-b66c-db0f5500c483-maas-api-openshift-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:41.818995 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:41.818940    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7lx6\" (UniqueName: \"kubernetes.io/projected/d500889f-a602-4b38-b66c-db0f5500c483-kube-api-access-z7lx6\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:41.818995 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:41.818968    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d500889f-a602-4b38-b66c-db0f5500c483-maas-api-xks-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:42.536171 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:42.536114    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf" event={"ID":"d500889f-a602-4b38-b66c-db0f5500c483","Type":"ContainerDied","Data":"1f6868f10bdc272d30780952c0657bfc98394da05f592988423faa3f182bc141"}
Oct 31 14:16:42.536383 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:42.536203    4428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f6868f10bdc272d30780952c0657bfc98394da05f592988423faa3f182bc141"
Oct 31 14:16:42.536383 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:42.536263    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"
Oct 31 14:16:42.754599 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:42.754548    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a239bc40-aa03-40ae-b7ff-c33ef400f867" path="/var/lib/kubelet/pods/a239bc40-aa03-40ae-b7ff-c33ef400f867/volumes"
Oct 31 14:16:48.740939 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:16:48.740895    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:16:50.534145 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.534109    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"]
Oct 31 14:16:50.534742 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.534719    4428 cpu_manager.go:414] "containerMap: removing container" podUID="d500889f-a602-4b38-b66c-db0f5500c483" containerName="revoke-keys"
Oct 31 14:16:50.534840 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.534745    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d500889f-a602-4b38-b66c-db0f5500c483" containerName="revoke-keys"
Oct 31 14:16:50.534929 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.534895    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d500889f-a602-4b38-b66c-db0f5500c483" containerName="revoke-keys"
Oct 31 14:16:50.539376 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.539352    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:50.550964 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.550940    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 14:16:50.555041 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.555011    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"]
Oct 31 14:16:50.558666 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.558634    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"
Oct 31 14:16:50.558806 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.558754    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"]
Oct 31 14:16:50.570641 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.570252    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"]
Oct 31 14:16:50.607523 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.607483    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfb4j\" (UniqueName: \"kubernetes.io/projected/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-kube-api-access-rfb4j\") pod \"maas-api-6d475f7bf9-p5p26\" (UID: \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:50.607756 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.607549    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-maas-api-tls\") pod \"maas-api-6d475f7bf9-p5p26\" (UID: \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:50.607756 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.607628    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stdf2\" (UniqueName: \"kubernetes.io/projected/1c4554c2-52c9-4417-9388-cff7dc8add81-kube-api-access-stdf2\") pod \"maas-api-64fbfb966c-6jndx\" (UID: \"1c4554c2-52c9-4417-9388-cff7dc8add81\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"
Oct 31 14:16:50.607756 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.607669    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4554c2-52c9-4417-9388-cff7dc8add81-openshift-service-ca\") pod \"maas-api-64fbfb966c-6jndx\" (UID: \"1c4554c2-52c9-4417-9388-cff7dc8add81\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"
Oct 31 14:16:50.607756 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.607690    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c4554c2-52c9-4417-9388-cff7dc8add81-maas-api-tls\") pod \"maas-api-64fbfb966c-6jndx\" (UID: \"1c4554c2-52c9-4417-9388-cff7dc8add81\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"
Oct 31 14:16:50.607756 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.607715    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c4554c2-52c9-4417-9388-cff7dc8add81-metrics-certs\") pod \"maas-api-64fbfb966c-6jndx\" (UID: \"1c4554c2-52c9-4417-9388-cff7dc8add81\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"
Oct 31 14:16:50.608001 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.607817    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-metrics-certs\") pod \"maas-api-6d475f7bf9-p5p26\" (UID: \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:50.608001 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.607887    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-openshift-service-ca\") pod \"maas-api-6d475f7bf9-p5p26\" (UID: \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:50.647468 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.647439    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bx4t6\""
Oct 31 14:16:50.647667 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.647511    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 14:16:50.709400 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.709359    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stdf2\" (UniqueName: \"kubernetes.io/projected/1c4554c2-52c9-4417-9388-cff7dc8add81-kube-api-access-stdf2\") pod \"maas-api-64fbfb966c-6jndx\" (UID: \"1c4554c2-52c9-4417-9388-cff7dc8add81\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"
Oct 31 14:16:50.709659 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.709432    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4554c2-52c9-4417-9388-cff7dc8add81-openshift-service-ca\") pod \"maas-api-64fbfb966c-6jndx\" (UID: \"1c4554c2-52c9-4417-9388-cff7dc8add81\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"
Oct 31 14:16:50.709659 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.709460    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c4554c2-52c9-4417-9388-cff7dc8add81-maas-api-tls\") pod \"maas-api-64fbfb966c-6jndx\" (UID: \"1c4554c2-52c9-4417-9388-cff7dc8add81\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"
Oct 31 14:16:50.709659 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.709495    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c4554c2-52c9-4417-9388-cff7dc8add81-metrics-certs\") pod \"maas-api-64fbfb966c-6jndx\" (UID: \"1c4554c2-52c9-4417-9388-cff7dc8add81\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"
Oct 31 14:16:50.709659 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.709523    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-metrics-certs\") pod \"maas-api-6d475f7bf9-p5p26\" (UID: \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:50.709659 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.709575    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-openshift-service-ca\") pod \"maas-api-6d475f7bf9-p5p26\" (UID: \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:50.709659 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.709629    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfb4j\" (UniqueName: \"kubernetes.io/projected/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-kube-api-access-rfb4j\") pod \"maas-api-6d475f7bf9-p5p26\" (UID: \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:50.710106 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.709682    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-maas-api-tls\") pod \"maas-api-6d475f7bf9-p5p26\" (UID: \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:50.710764 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.710732    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4554c2-52c9-4417-9388-cff7dc8add81-openshift-service-ca\") pod \"maas-api-64fbfb966c-6jndx\" (UID: \"1c4554c2-52c9-4417-9388-cff7dc8add81\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"
Oct 31 14:16:50.710938 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.710795    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-openshift-service-ca\") pod \"maas-api-6d475f7bf9-p5p26\" (UID: \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:50.712556 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.712531    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-maas-api-tls\") pod \"maas-api-6d475f7bf9-p5p26\" (UID: \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:50.712685 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.712607    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c4554c2-52c9-4417-9388-cff7dc8add81-maas-api-tls\") pod \"maas-api-64fbfb966c-6jndx\" (UID: \"1c4554c2-52c9-4417-9388-cff7dc8add81\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"
Oct 31 14:16:50.713219 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.713190    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c4554c2-52c9-4417-9388-cff7dc8add81-metrics-certs\") pod \"maas-api-64fbfb966c-6jndx\" (UID: \"1c4554c2-52c9-4417-9388-cff7dc8add81\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"
Oct 31 14:16:50.713517 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.713487    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-metrics-certs\") pod \"maas-api-6d475f7bf9-p5p26\" (UID: \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:50.724133 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.724110    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stdf2\" (UniqueName: \"kubernetes.io/projected/1c4554c2-52c9-4417-9388-cff7dc8add81-kube-api-access-stdf2\") pod \"maas-api-64fbfb966c-6jndx\" (UID: \"1c4554c2-52c9-4417-9388-cff7dc8add81\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"
Oct 31 14:16:50.724274 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.724228    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfb4j\" (UniqueName: \"kubernetes.io/projected/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-kube-api-access-rfb4j\") pod \"maas-api-6d475f7bf9-p5p26\" (UID: \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\") " pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:50.855014 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.854919    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:50.879533 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:50.879503    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"
Oct 31 14:16:51.149593 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:51.149555    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"]
Oct 31 14:16:51.151817 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:16:51.151785    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c7872f_eab6_4c46_ae4c_aa9964ab6cde.slice/crio-51025d412a063980a85096928b36a6cad3b3e0b76549a0eff3a03c9d30edfe46 WatchSource:0}: Error finding container 51025d412a063980a85096928b36a6cad3b3e0b76549a0eff3a03c9d30edfe46: Status 404 returned error can't find the container with id 51025d412a063980a85096928b36a6cad3b3e0b76549a0eff3a03c9d30edfe46
Oct 31 14:16:51.153470 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:51.153449    4428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 14:16:51.176641 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:51.176616    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"]
Oct 31 14:16:51.178844 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:16:51.178817    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4554c2_52c9_4417_9388_cff7dc8add81.slice/crio-f5280ed115d99b9e6ec354ea05972483415208d8cbeb52995079354cbad5cc76 WatchSource:0}: Error finding container f5280ed115d99b9e6ec354ea05972483415208d8cbeb52995079354cbad5cc76: Status 404 returned error can't find the container with id f5280ed115d99b9e6ec354ea05972483415208d8cbeb52995079354cbad5cc76
Oct 31 14:16:51.586795 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:51.586744    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx" event={"ID":"1c4554c2-52c9-4417-9388-cff7dc8add81","Type":"ContainerStarted","Data":"f5280ed115d99b9e6ec354ea05972483415208d8cbeb52995079354cbad5cc76"}
Oct 31 14:16:51.588330 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:51.588292    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26" event={"ID":"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde","Type":"ContainerStarted","Data":"51025d412a063980a85096928b36a6cad3b3e0b76549a0eff3a03c9d30edfe46"}
Oct 31 14:16:52.595861 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:52.595817    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx" event={"ID":"1c4554c2-52c9-4417-9388-cff7dc8add81","Type":"ContainerStarted","Data":"f2f97c7b6ea757dbaedc1a5d353de929e134a6d27354cd4bf249d45c2e5b58e1"}
Oct 31 14:16:52.596371 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:52.595923    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"
Oct 31 14:16:52.597612 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:52.597585    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26" event={"ID":"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde","Type":"ContainerStarted","Data":"3ba9004efc192e239286abc194d71ace265250f8d08962fa9b669d609db3880e"}
Oct 31 14:16:52.597727 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:52.597702    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:52.619848 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:52.619728    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx" podStartSLOduration=2.095272966 podStartE2EDuration="2.619697818s" totalImagesPullingTime="524.424852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:16:52.617421579 +0000 UTC m=+4410.015770204" watchObservedRunningTime="2026-10-31 14:16:52.619697818 +0000 UTC m=+4410.018046441"
Oct 31 14:16:52.646423 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:52.646354    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26" podStartSLOduration=2.078153598 podStartE2EDuration="2.646334996s" totalImagesPullingTime="568.181398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:16:52.642225421 +0000 UTC m=+4410.040574046" watchObservedRunningTime="2026-10-31 14:16:52.646334996 +0000 UTC m=+4410.044683622"
Oct 31 14:16:57.580163 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:57.580127    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:16:57.970463 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:57.970425    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"]
Oct 31 14:16:57.973947 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:57.973924    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:57.980402 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:57.980378    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a1008181-openshift-default-dockercfg-pdnhv\""
Oct 31 14:16:57.996825 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:57.996793    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"]
Oct 31 14:16:58.082637 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.082596    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd571eff-4b33-473e-9994-a4b1328a066c-istiod-ca-cert\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.082858 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.082649    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-credential-socket\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.082858 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.082730    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb6f8\" (UniqueName: \"kubernetes.io/projected/bd571eff-4b33-473e-9994-a4b1328a066c-kube-api-access-pb6f8\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.082858 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.082759    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-istio-data\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.082858 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.082789    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-istio-envoy\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.083045 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.082974    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-workload-socket\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.083045 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.083020    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd571eff-4b33-473e-9994-a4b1328a066c-istio-token\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.083124 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.083057    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-workload-certs\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.083124 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.083099    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd571eff-4b33-473e-9994-a4b1328a066c-istio-podinfo\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.184617 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.184571    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-credential-socket\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.184617 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.184621    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb6f8\" (UniqueName: \"kubernetes.io/projected/bd571eff-4b33-473e-9994-a4b1328a066c-kube-api-access-pb6f8\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.184925 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.184643    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-istio-data\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.184925 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.184671    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-istio-envoy\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.184925 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.184708    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-workload-socket\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.184925 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.184731    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd571eff-4b33-473e-9994-a4b1328a066c-istio-token\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.184925 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.184764    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-workload-certs\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.184925 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.184808    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd571eff-4b33-473e-9994-a4b1328a066c-istio-podinfo\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.185210 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.184958    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd571eff-4b33-473e-9994-a4b1328a066c-istiod-ca-cert\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.185210 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.185052    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-credential-socket\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.185210 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.185097    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-istio-data\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.185382 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.185336    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-workload-socket\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.185528 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.185502    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-workload-certs\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.185683 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.185660    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd571eff-4b33-473e-9994-a4b1328a066c-istiod-ca-cert\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.187593 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.187571    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-istio-envoy\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.187808 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.187786    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd571eff-4b33-473e-9994-a4b1328a066c-istio-podinfo\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.210997 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.210964    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd571eff-4b33-473e-9994-a4b1328a066c-istio-token\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.211607 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.211577    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb6f8\" (UniqueName: \"kubernetes.io/projected/bd571eff-4b33-473e-9994-a4b1328a066c-kube-api-access-pb6f8\") pod \"e2e-ait-a1008181-openshift-default-57476b49cc-b4htz\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") " pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.289427 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.289336    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:16:58.332436 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.332404    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:16:58.507698 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.507651    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:16:58.584993 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.584963    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"]
Oct 31 14:16:58.586530 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:16:58.586491    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd571eff_4b33_473e_9994_a4b1328a066c.slice/crio-6d3793d20c8f97620bd3601d777ac32ba08d1c95a9b0430015ba4aea7e3c3653 WatchSource:0}: Error finding container 6d3793d20c8f97620bd3601d777ac32ba08d1c95a9b0430015ba4aea7e3c3653: Status 404 returned error can't find the container with id 6d3793d20c8f97620bd3601d777ac32ba08d1c95a9b0430015ba4aea7e3c3653
Oct 31 14:16:58.589038 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.589002    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:16:58.589130 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.589079    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:16:58.589130 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.589110    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:16:58.610293 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.610261    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:58.610649 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.610627    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-64fbfb966c-6jndx"
Oct 31 14:16:58.630022 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.629988    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz" event={"ID":"bd571eff-4b33-473e-9994-a4b1328a066c","Type":"ContainerStarted","Data":"6d3793d20c8f97620bd3601d777ac32ba08d1c95a9b0430015ba4aea7e3c3653"}
Oct 31 14:16:58.693857 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.693819    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"]
Oct 31 14:16:58.694411 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.694369    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26" podUID="e6c7872f-eab6-4c46-ae4c-aa9964ab6cde" containerName="maas-api" containerID="cri-o://3ba9004efc192e239286abc194d71ace265250f8d08962fa9b669d609db3880e" gracePeriod=30
Oct 31 14:16:58.956501 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:58.956470    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:59.095755 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.095720    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-kube-api-access-rfb4j\" (UniqueName: \"kubernetes.io/projected/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-kube-api-access-rfb4j\") pod \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\" (UID: \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\") "
Oct 31 14:16:59.096008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.095796    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-metrics-certs\") pod \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\" (UID: \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\") "
Oct 31 14:16:59.096008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.095832    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-maas-api-tls\") pod \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\" (UID: \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\") "
Oct 31 14:16:59.096008 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.095933    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-openshift-service-ca\") pod \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\" (UID: \"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde\") "
Oct 31 14:16:59.096380 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.096352    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-openshift-service-ca" pod "e6c7872f-eab6-4c46-ae4c-aa9964ab6cde" (UID: "e6c7872f-eab6-4c46-ae4c-aa9964ab6cde"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:16:59.098237 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.098203    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-kube-api-access-rfb4j" pod "e6c7872f-eab6-4c46-ae4c-aa9964ab6cde" (UID: "e6c7872f-eab6-4c46-ae4c-aa9964ab6cde"). InnerVolumeSpecName "kube-api-access-rfb4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:16:59.098743 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.098721    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-metrics-certs" pod "e6c7872f-eab6-4c46-ae4c-aa9964ab6cde" (UID: "e6c7872f-eab6-4c46-ae4c-aa9964ab6cde"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:16:59.098830 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.098741    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-maas-api-tls" pod "e6c7872f-eab6-4c46-ae4c-aa9964ab6cde" (UID: "e6c7872f-eab6-4c46-ae4c-aa9964ab6cde"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:16:59.197154 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.197119    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:59.197154 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.197154    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:59.197379 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.197167    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:59.197379 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.197181    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfb4j\" (UniqueName: \"kubernetes.io/projected/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde-kube-api-access-rfb4j\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:16:59.637375 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.637336    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz" event={"ID":"bd571eff-4b33-473e-9994-a4b1328a066c","Type":"ContainerStarted","Data":"36778f36963aedaf5a88f8fdefe87a47b5229714ebc34db7dcdc9d0f5f625a00"}
Oct 31 14:16:59.639193 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.639162    4428 generic.go:356] "Generic (PLEG): container finished" podID="e6c7872f-eab6-4c46-ae4c-aa9964ab6cde" containerID="3ba9004efc192e239286abc194d71ace265250f8d08962fa9b669d609db3880e" exitCode=0
Oct 31 14:16:59.639375 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.639221    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"
Oct 31 14:16:59.639375 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.639248    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26" event={"ID":"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde","Type":"ContainerDied","Data":"3ba9004efc192e239286abc194d71ace265250f8d08962fa9b669d609db3880e"}
Oct 31 14:16:59.639375 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.639300    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26" event={"ID":"e6c7872f-eab6-4c46-ae4c-aa9964ab6cde","Type":"ContainerDied","Data":"51025d412a063980a85096928b36a6cad3b3e0b76549a0eff3a03c9d30edfe46"}
Oct 31 14:16:59.639375 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.639331    4428 scope.go:122] "RemoveContainer" containerID="3ba9004efc192e239286abc194d71ace265250f8d08962fa9b669d609db3880e"
Oct 31 14:16:59.651102 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.651081    4428 scope.go:122] "RemoveContainer" containerID="3ba9004efc192e239286abc194d71ace265250f8d08962fa9b669d609db3880e"
Oct 31 14:16:59.651446 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:16:59.651425    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba9004efc192e239286abc194d71ace265250f8d08962fa9b669d609db3880e\": container with ID starting with 3ba9004efc192e239286abc194d71ace265250f8d08962fa9b669d609db3880e not found: ID does not exist" containerID="3ba9004efc192e239286abc194d71ace265250f8d08962fa9b669d609db3880e"
Oct 31 14:16:59.651510 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.651455    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba9004efc192e239286abc194d71ace265250f8d08962fa9b669d609db3880e"} err="failed to get container status \"3ba9004efc192e239286abc194d71ace265250f8d08962fa9b669d609db3880e\": rpc error: code = NotFound desc = could not find container \"3ba9004efc192e239286abc194d71ace265250f8d08962fa9b669d609db3880e\": container with ID starting with 3ba9004efc192e239286abc194d71ace265250f8d08962fa9b669d609db3880e not found: ID does not exist"
Oct 31 14:16:59.665269 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.665226    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz" podStartSLOduration=2.66521222 podStartE2EDuration="2.66521222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:16:59.663003619 +0000 UTC m=+4417.061352244" watchObservedRunningTime="2026-10-31 14:16:59.66521222 +0000 UTC m=+4417.063560843"
Oct 31 14:16:59.679412 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.679374    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"]
Oct 31 14:16:59.683847 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:16:59.683819    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d475f7bf9-p5p26"]
Oct 31 14:17:00.290543 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:00.290500    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:17:00.295853 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:00.295824    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:17:00.646956 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:00.646839    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:17:00.648056 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:00.648032    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:17:00.755684 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:00.755643    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6c7872f-eab6-4c46-ae4c-aa9964ab6cde" path="/var/lib/kubelet/pods/e6c7872f-eab6-4c46-ae4c-aa9964ab6cde/volumes"
Oct 31 14:17:01.213215 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:01.213179    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:01.376490 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:01.376459    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:02.421287 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.421251    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"]
Oct 31 14:17:02.421739 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.421685    4428 cpu_manager.go:414] "containerMap: removing container" podUID="e6c7872f-eab6-4c46-ae4c-aa9964ab6cde" containerName="maas-api"
Oct 31 14:17:02.421739 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.421696    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c7872f-eab6-4c46-ae4c-aa9964ab6cde" containerName="maas-api"
Oct 31 14:17:02.421834 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.421778    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c7872f-eab6-4c46-ae4c-aa9964ab6cde" containerName="maas-api"
Oct 31 14:17:02.425254 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.425229    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:02.430078 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.430052    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a1008181-serving-cert\""
Oct 31 14:17:02.442146 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.442112    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:02.450797 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.450756    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"]
Oct 31 14:17:02.451027 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.450950    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"]
Oct 31 14:17:02.456154 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.456119    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"
Oct 31 14:17:02.461694 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.461645    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a1008181\""
Oct 31 14:17:02.468434 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.468404    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"]
Oct 31 14:17:02.484445 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.484418    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"]
Oct 31 14:17:02.488386 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.488356    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"
Oct 31 14:17:02.503810 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.503780    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"]
Oct 31 14:17:02.531128 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.531086    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a1008181-dockercfg-pv97p\""
Oct 31 14:17:02.538521 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.538488    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7ecbf14-9916-42a1-84bf-0945115f7d24-maas-api-tls\") pod \"maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l\" (UID: \"b7ecbf14-9916-42a1-84bf-0945115f7d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:02.538724 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.538547    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7ecbf14-9916-42a1-84bf-0945115f7d24-openshift-service-ca\") pod \"maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l\" (UID: \"b7ecbf14-9916-42a1-84bf-0945115f7d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:02.538819 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.538725    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt6pv\" (UniqueName: \"kubernetes.io/projected/b7ecbf14-9916-42a1-84bf-0945115f7d24-kube-api-access-kt6pv\") pod \"maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l\" (UID: \"b7ecbf14-9916-42a1-84bf-0945115f7d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:02.538819 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.538780    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7ecbf14-9916-42a1-84bf-0945115f7d24-metrics-certs\") pod \"maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l\" (UID: \"b7ecbf14-9916-42a1-84bf-0945115f7d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:02.617309 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.617273    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:02.639607 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.639566    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7457057f-33a8-42cc-a099-43796099665f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6\" (UID: \"7457057f-33a8-42cc-a099-43796099665f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"
Oct 31 14:17:02.639823 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.639624    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7ecbf14-9916-42a1-84bf-0945115f7d24-openshift-service-ca\") pod \"maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l\" (UID: \"b7ecbf14-9916-42a1-84bf-0945115f7d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:02.639823 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.639747    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt6pv\" (UniqueName: \"kubernetes.io/projected/b7ecbf14-9916-42a1-84bf-0945115f7d24-kube-api-access-kt6pv\") pod \"maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l\" (UID: \"b7ecbf14-9916-42a1-84bf-0945115f7d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:02.639823 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.639776    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqjm7\" (UniqueName: \"kubernetes.io/projected/5b24cdad-99eb-488e-8451-3beff4f272f4-kube-api-access-nqjm7\") pod \"payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs\" (UID: \"5b24cdad-99eb-488e-8451-3beff4f272f4\") " pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"
Oct 31 14:17:02.639823 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.639810    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7ecbf14-9916-42a1-84bf-0945115f7d24-metrics-certs\") pod \"maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l\" (UID: \"b7ecbf14-9916-42a1-84bf-0945115f7d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:02.640080 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.639898    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54vdm\" (UniqueName: \"kubernetes.io/projected/7457057f-33a8-42cc-a099-43796099665f-kube-api-access-54vdm\") pod \"payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6\" (UID: \"7457057f-33a8-42cc-a099-43796099665f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"
Oct 31 14:17:02.640080 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.639948    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7ecbf14-9916-42a1-84bf-0945115f7d24-maas-api-tls\") pod \"maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l\" (UID: \"b7ecbf14-9916-42a1-84bf-0945115f7d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:02.640204 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.640076    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b24cdad-99eb-488e-8451-3beff4f272f4-plugins-config-volume\") pod \"payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs\" (UID: \"5b24cdad-99eb-488e-8451-3beff4f272f4\") " pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"
Oct 31 14:17:02.640442 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.640424    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7ecbf14-9916-42a1-84bf-0945115f7d24-openshift-service-ca\") pod \"maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l\" (UID: \"b7ecbf14-9916-42a1-84bf-0945115f7d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:02.642540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.642518    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7ecbf14-9916-42a1-84bf-0945115f7d24-metrics-certs\") pod \"maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l\" (UID: \"b7ecbf14-9916-42a1-84bf-0945115f7d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:02.642629 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.642559    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7ecbf14-9916-42a1-84bf-0945115f7d24-maas-api-tls\") pod \"maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l\" (UID: \"b7ecbf14-9916-42a1-84bf-0945115f7d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:02.657681 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.657650    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt6pv\" (UniqueName: \"kubernetes.io/projected/b7ecbf14-9916-42a1-84bf-0945115f7d24-kube-api-access-kt6pv\") pod \"maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l\" (UID: \"b7ecbf14-9916-42a1-84bf-0945115f7d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:02.741529 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.741495    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqjm7\" (UniqueName: \"kubernetes.io/projected/5b24cdad-99eb-488e-8451-3beff4f272f4-kube-api-access-nqjm7\") pod \"payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs\" (UID: \"5b24cdad-99eb-488e-8451-3beff4f272f4\") " pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"
Oct 31 14:17:02.741758 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.741557    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54vdm\" (UniqueName: \"kubernetes.io/projected/7457057f-33a8-42cc-a099-43796099665f-kube-api-access-54vdm\") pod \"payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6\" (UID: \"7457057f-33a8-42cc-a099-43796099665f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"
Oct 31 14:17:02.741758 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.741591    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b24cdad-99eb-488e-8451-3beff4f272f4-plugins-config-volume\") pod \"payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs\" (UID: \"5b24cdad-99eb-488e-8451-3beff4f272f4\") " pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"
Oct 31 14:17:02.741758 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.741624    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7457057f-33a8-42cc-a099-43796099665f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6\" (UID: \"7457057f-33a8-42cc-a099-43796099665f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"
Oct 31 14:17:02.742438 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.742414    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b24cdad-99eb-488e-8451-3beff4f272f4-plugins-config-volume\") pod \"payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs\" (UID: \"5b24cdad-99eb-488e-8451-3beff4f272f4\") " pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"
Oct 31 14:17:02.742503 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.742422    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7457057f-33a8-42cc-a099-43796099665f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6\" (UID: \"7457057f-33a8-42cc-a099-43796099665f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"
Oct 31 14:17:02.751537 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.751508    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:02.755390 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.755360    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqjm7\" (UniqueName: \"kubernetes.io/projected/5b24cdad-99eb-488e-8451-3beff4f272f4-kube-api-access-nqjm7\") pod \"payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs\" (UID: \"5b24cdad-99eb-488e-8451-3beff4f272f4\") " pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"
Oct 31 14:17:02.755536 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.755514    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54vdm\" (UniqueName: \"kubernetes.io/projected/7457057f-33a8-42cc-a099-43796099665f-kube-api-access-54vdm\") pod \"payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6\" (UID: \"7457057f-33a8-42cc-a099-43796099665f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"
Oct 31 14:17:02.773732 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.773706    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"
Oct 31 14:17:02.792117 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.792077    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:02.801465 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:02.801438    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"
Oct 31 14:17:03.062513 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:03.062462    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"]
Oct 31 14:17:03.065212 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:17:03.065181    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ecbf14_9916_42a1_84bf_0945115f7d24.slice/crio-dfdd84dbb32016e1d31fed5a617280fc178d8af45c6700cb417c22faf442ecfe WatchSource:0}: Error finding container dfdd84dbb32016e1d31fed5a617280fc178d8af45c6700cb417c22faf442ecfe: Status 404 returned error can't find the container with id dfdd84dbb32016e1d31fed5a617280fc178d8af45c6700cb417c22faf442ecfe
Oct 31 14:17:03.082917 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:17:03.082858    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7457057f_33a8_42cc_a099_43796099665f.slice/crio-e93c4dcc001a2b1d5e8d2fd71454f07f573432613a72b1e835585b5fec73b105 WatchSource:0}: Error finding container e93c4dcc001a2b1d5e8d2fd71454f07f573432613a72b1e835585b5fec73b105: Status 404 returned error can't find the container with id e93c4dcc001a2b1d5e8d2fd71454f07f573432613a72b1e835585b5fec73b105
Oct 31 14:17:03.085131 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:03.085105    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"]
Oct 31 14:17:03.126337 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:03.126304    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"]
Oct 31 14:17:03.130846 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:17:03.130813    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b24cdad_99eb_488e_8451_3beff4f272f4.slice/crio-d2467a593cd7fb387745c781d1f2694013d52d55c4214065e1f732b868b4753d WatchSource:0}: Error finding container d2467a593cd7fb387745c781d1f2694013d52d55c4214065e1f732b868b4753d: Status 404 returned error can't find the container with id d2467a593cd7fb387745c781d1f2694013d52d55c4214065e1f732b868b4753d
Oct 31 14:17:03.553191 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:03.553157    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:03.664261 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:03.664220    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l" event={"ID":"b7ecbf14-9916-42a1-84bf-0945115f7d24","Type":"ContainerStarted","Data":"b70dc5e49d4fc00c75657a4812aff8c3a60021021edef6a13d74ae0871fafaa0"}
Oct 31 14:17:03.664462 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:03.664293    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:03.664462 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:03.664320    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l" event={"ID":"b7ecbf14-9916-42a1-84bf-0945115f7d24","Type":"ContainerStarted","Data":"dfdd84dbb32016e1d31fed5a617280fc178d8af45c6700cb417c22faf442ecfe"}
Oct 31 14:17:03.666023 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:03.665995    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6" event={"ID":"7457057f-33a8-42cc-a099-43796099665f","Type":"ContainerStarted","Data":"318e99ab7777e91ba20ba7416c5b9f51a325ce35b41a05979a297444db7f40be"}
Oct 31 14:17:03.666155 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:03.666072    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6" event={"ID":"7457057f-33a8-42cc-a099-43796099665f","Type":"ContainerStarted","Data":"e93c4dcc001a2b1d5e8d2fd71454f07f573432613a72b1e835585b5fec73b105"}
Oct 31 14:17:03.666155 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:03.666094    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"
Oct 31 14:17:03.667678 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:03.667655    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs" event={"ID":"5b24cdad-99eb-488e-8451-3beff4f272f4","Type":"ContainerStarted","Data":"453ff708c1c1539a7cd76f8f936c153f65ecfa5c2d5fd27c54ee2d679176e895"}
Oct 31 14:17:03.667816 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:03.667699    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs" event={"ID":"5b24cdad-99eb-488e-8451-3beff4f272f4","Type":"ContainerStarted","Data":"d2467a593cd7fb387745c781d1f2694013d52d55c4214065e1f732b868b4753d"}
Oct 31 14:17:03.667816 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:03.667778    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"
Oct 31 14:17:03.686282 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:03.686236    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l" podStartSLOduration=1.315674496 podStartE2EDuration="1.686222158s" totalImagesPullingTime="370.547662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:17:03.684944423 +0000 UTC m=+4421.083293048" watchObservedRunningTime="2026-10-31 14:17:03.686222158 +0000 UTC m=+4421.084570782"
Oct 31 14:17:03.709497 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:03.709382    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6" podStartSLOduration=1.709368074 podStartE2EDuration="1.709368074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:17:03.707834002 +0000 UTC m=+4421.106182629" watchObservedRunningTime="2026-10-31 14:17:03.709368074 +0000 UTC m=+4421.107716697"
Oct 31 14:17:03.726861 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:03.726810    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs" podStartSLOduration=1.7267949040000001 podStartE2EDuration="1.726794904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:17:03.725919773 +0000 UTC m=+4421.124268399" watchObservedRunningTime="2026-10-31 14:17:03.726794904 +0000 UTC m=+4421.125143528"
Oct 31 14:17:04.429282 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:04.429244    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:05.136858 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:05.136826    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:06.015183 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:06.015149    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:06.769895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:06.769433    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:07.542929 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:07.542886    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:08.325132 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:08.325074    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:09.466123 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:09.466081    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:09.679732 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:09.679704    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:09.879338 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:09.879250    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:10.726704 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:10.726668    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:11.589755 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:11.589723    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:12.563822 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:12.563779    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:13.350578 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:13.350548    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:14.238047 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:14.238012    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:14.676151 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:14.676126    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"
Oct 31 14:17:14.676509 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:14.676493    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"
Oct 31 14:17:14.734206 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:14.734167    4428 generic.go:356] "Generic (PLEG): container finished" podID="bd571eff-4b33-473e-9994-a4b1328a066c" containerID="36778f36963aedaf5a88f8fdefe87a47b5229714ebc34db7dcdc9d0f5f625a00" exitCode=0
Oct 31 14:17:14.734400 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:14.734325    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz" event={"ID":"bd571eff-4b33-473e-9994-a4b1328a066c","Type":"ContainerDied","Data":"36778f36963aedaf5a88f8fdefe87a47b5229714ebc34db7dcdc9d0f5f625a00"}
Oct 31 14:17:14.734894 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:14.734844    4428 scope.go:122] "RemoveContainer" containerID="36778f36963aedaf5a88f8fdefe87a47b5229714ebc34db7dcdc9d0f5f625a00"
Oct 31 14:17:14.736529 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:14.736491    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:17:14.736649 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:14.736594    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:17:14.736706 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:14.736648    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:17:15.051610 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:15.051574    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:15.668896 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:15.654966    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:17:15.741810 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:15.741774    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz" event={"ID":"bd571eff-4b33-473e-9994-a4b1328a066c","Type":"ContainerStarted","Data":"b45b9cc70fad4992af69254477c2740818a18006bcecf3ae3ae54ea37cde8061"}
Oct 31 14:17:15.822491 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:15.822452    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:16.290526 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:16.290492    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:17:16.291569 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:16.291542    4428 patch_prober.go:29] interesting pod/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused" start-of-body=
Oct 31 14:17:16.291711 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:16.291606    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz" podUID="bd571eff-4b33-473e-9994-a4b1328a066c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused"
Oct 31 14:17:16.611934 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:16.611819    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:16.771525 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:16.771486    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:17.290064 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:17.290027    4428 patch_prober.go:29] interesting pod/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused" start-of-body=
Oct 31 14:17:17.290280 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:17.290089    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz" podUID="bd571eff-4b33-473e-9994-a4b1328a066c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused"
Oct 31 14:17:17.368895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:17.367160    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:17.529461 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:17.529428    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:18.289897 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:18.289830    4428 patch_prober.go:29] interesting pod/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused" start-of-body=
Oct 31 14:17:18.290342 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:18.289924    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz" podUID="bd571eff-4b33-473e-9994-a4b1328a066c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused"
Oct 31 14:17:18.338925 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:18.338843    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:18.740988 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:17:18.740949    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:17:19.063956 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:19.060902    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:19.289764 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:19.289724    4428 patch_prober.go:29] interesting pod/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused" start-of-body=
Oct 31 14:17:19.290025 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:19.289801    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz" podUID="bd571eff-4b33-473e-9994-a4b1328a066c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused"
Oct 31 14:17:19.842641 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:19.842601    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:20.290522 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:20.290482    4428 patch_prober.go:29] interesting pod/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused" start-of-body=
Oct 31 14:17:20.290970 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:20.290544    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz" podUID="bd571eff-4b33-473e-9994-a4b1328a066c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.96:15021/healthz/ready\": dial tcp 10.134.1.96:15021: connect: connection refused"
Oct 31 14:17:20.640261 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:20.640173    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:21.294171 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:21.294141    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:17:21.294651 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:21.294599    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:17:21.295495 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:21.295478    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:17:21.356821 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:21.356776    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:22.215426 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:22.215391    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:22.846996 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:22.846959    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:23.676744 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:23.676711    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:24.438278 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:24.438247    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:25.153166 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:25.153137    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:25.844049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:25.844017    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:26.641986 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:26.641951    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:27.221780 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:27.221746    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"]
Oct 31 14:17:27.238233 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:27.238188    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"]
Oct 31 14:17:27.353494 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:27.353463    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:27.511981 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:27.511857    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:27.882301 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:27.882174    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"]
Oct 31 14:17:27.882575 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:27.882475    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l" podUID="b7ecbf14-9916-42a1-84bf-0945115f7d24" containerName="maas-api" containerID="cri-o://b70dc5e49d4fc00c75657a4812aff8c3a60021021edef6a13d74ae0871fafaa0" gracePeriod=30
Oct 31 14:17:27.996862 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:27.996747    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:28.163089 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.163056    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:28.193567 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.193498    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:28.210296 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.210260    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ecbf14-9916-42a1-84bf-0945115f7d24-kube-api-access-kt6pv\" (UniqueName: \"kubernetes.io/projected/b7ecbf14-9916-42a1-84bf-0945115f7d24-kube-api-access-kt6pv\") pod \"b7ecbf14-9916-42a1-84bf-0945115f7d24\" (UID: \"b7ecbf14-9916-42a1-84bf-0945115f7d24\") "
Oct 31 14:17:28.210481 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.210372    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7ecbf14-9916-42a1-84bf-0945115f7d24-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7ecbf14-9916-42a1-84bf-0945115f7d24-metrics-certs\") pod \"b7ecbf14-9916-42a1-84bf-0945115f7d24\" (UID: \"b7ecbf14-9916-42a1-84bf-0945115f7d24\") "
Oct 31 14:17:28.210540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.210498    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7ecbf14-9916-42a1-84bf-0945115f7d24-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7ecbf14-9916-42a1-84bf-0945115f7d24-maas-api-tls\") pod \"b7ecbf14-9916-42a1-84bf-0945115f7d24\" (UID: \"b7ecbf14-9916-42a1-84bf-0945115f7d24\") "
Oct 31 14:17:28.210599 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.210550    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7ecbf14-9916-42a1-84bf-0945115f7d24-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7ecbf14-9916-42a1-84bf-0945115f7d24-openshift-service-ca\") pod \"b7ecbf14-9916-42a1-84bf-0945115f7d24\" (UID: \"b7ecbf14-9916-42a1-84bf-0945115f7d24\") "
Oct 31 14:17:28.211395 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.211361    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7ecbf14-9916-42a1-84bf-0945115f7d24-openshift-service-ca" pod "b7ecbf14-9916-42a1-84bf-0945115f7d24" (UID: "b7ecbf14-9916-42a1-84bf-0945115f7d24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:17:28.212935 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.212907    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7ecbf14-9916-42a1-84bf-0945115f7d24-metrics-certs" pod "b7ecbf14-9916-42a1-84bf-0945115f7d24" (UID: "b7ecbf14-9916-42a1-84bf-0945115f7d24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:17:28.213272 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.213248    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ecbf14-9916-42a1-84bf-0945115f7d24-kube-api-access-kt6pv" pod "b7ecbf14-9916-42a1-84bf-0945115f7d24" (UID: "b7ecbf14-9916-42a1-84bf-0945115f7d24"). InnerVolumeSpecName "kube-api-access-kt6pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:17:28.213433 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.213408    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7ecbf14-9916-42a1-84bf-0945115f7d24-maas-api-tls" pod "b7ecbf14-9916-42a1-84bf-0945115f7d24" (UID: "b7ecbf14-9916-42a1-84bf-0945115f7d24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:17:28.311941 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.311902    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7ecbf14-9916-42a1-84bf-0945115f7d24-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:28.311941 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.311937    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7ecbf14-9916-42a1-84bf-0945115f7d24-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:28.311941 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.311947    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7ecbf14-9916-42a1-84bf-0945115f7d24-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:28.312456 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.311958    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt6pv\" (UniqueName: \"kubernetes.io/projected/b7ecbf14-9916-42a1-84bf-0945115f7d24-kube-api-access-kt6pv\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:28.350573 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.350540    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:28.839899 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.839835    4428 generic.go:356] "Generic (PLEG): container finished" podID="b7ecbf14-9916-42a1-84bf-0945115f7d24" containerID="b70dc5e49d4fc00c75657a4812aff8c3a60021021edef6a13d74ae0871fafaa0" exitCode=0
Oct 31 14:17:28.840117 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.839901    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l" event={"ID":"b7ecbf14-9916-42a1-84bf-0945115f7d24","Type":"ContainerDied","Data":"b70dc5e49d4fc00c75657a4812aff8c3a60021021edef6a13d74ae0871fafaa0"}
Oct 31 14:17:28.840117 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.839959    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l" event={"ID":"b7ecbf14-9916-42a1-84bf-0945115f7d24","Type":"ContainerDied","Data":"dfdd84dbb32016e1d31fed5a617280fc178d8af45c6700cb417c22faf442ecfe"}
Oct 31 14:17:28.840117 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.839960    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"
Oct 31 14:17:28.840117 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.840026    4428 scope.go:122] "RemoveContainer" containerID="b70dc5e49d4fc00c75657a4812aff8c3a60021021edef6a13d74ae0871fafaa0"
Oct 31 14:17:28.849779 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.849749    4428 scope.go:122] "RemoveContainer" containerID="b70dc5e49d4fc00c75657a4812aff8c3a60021021edef6a13d74ae0871fafaa0"
Oct 31 14:17:28.850079 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:17:28.850057    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70dc5e49d4fc00c75657a4812aff8c3a60021021edef6a13d74ae0871fafaa0\": container with ID starting with b70dc5e49d4fc00c75657a4812aff8c3a60021021edef6a13d74ae0871fafaa0 not found: ID does not exist" containerID="b70dc5e49d4fc00c75657a4812aff8c3a60021021edef6a13d74ae0871fafaa0"
Oct 31 14:17:28.850170 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.850090    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70dc5e49d4fc00c75657a4812aff8c3a60021021edef6a13d74ae0871fafaa0"} err="failed to get container status \"b70dc5e49d4fc00c75657a4812aff8c3a60021021edef6a13d74ae0871fafaa0\": rpc error: code = NotFound desc = could not find container \"b70dc5e49d4fc00c75657a4812aff8c3a60021021edef6a13d74ae0871fafaa0\": container with ID starting with b70dc5e49d4fc00c75657a4812aff8c3a60021021edef6a13d74ae0871fafaa0 not found: ID does not exist"
Oct 31 14:17:28.859480 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.859454    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"]
Oct 31 14:17:28.865024 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:28.865004    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1008181-67dc8c88c-5jw5l"]
Oct 31 14:17:30.755692 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:30.755655    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7ecbf14-9916-42a1-84bf-0945115f7d24" path="/var/lib/kubelet/pods/b7ecbf14-9916-42a1-84bf-0945115f7d24/volumes"
Oct 31 14:17:32.222840 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.222780    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs" podUID="5b24cdad-99eb-488e-8451-3beff4f272f4" containerName="payload-processing" containerID="cri-o://453ff708c1c1539a7cd76f8f936c153f65ecfa5c2d5fd27c54ee2d679176e895" gracePeriod=25
Oct 31 14:17:32.239496 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.239430    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6" podUID="7457057f-33a8-42cc-a099-43796099665f" containerName="payload-pre-processing" containerID="cri-o://318e99ab7777e91ba20ba7416c5b9f51a325ce35b41a05979a297444db7f40be" gracePeriod=25
Oct 31 14:17:32.516314 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.516287    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"
Oct 31 14:17:32.543709 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.543679    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"
Oct 31 14:17:32.655084 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.655045    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7457057f-33a8-42cc-a099-43796099665f-kube-api-access-54vdm\" (UniqueName: \"kubernetes.io/projected/7457057f-33a8-42cc-a099-43796099665f-kube-api-access-54vdm\") pod \"7457057f-33a8-42cc-a099-43796099665f\" (UID: \"7457057f-33a8-42cc-a099-43796099665f\") "
Oct 31 14:17:32.655084 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.655089    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b24cdad-99eb-488e-8451-3beff4f272f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b24cdad-99eb-488e-8451-3beff4f272f4-plugins-config-volume\") pod \"5b24cdad-99eb-488e-8451-3beff4f272f4\" (UID: \"5b24cdad-99eb-488e-8451-3beff4f272f4\") "
Oct 31 14:17:32.655394 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.655115    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b24cdad-99eb-488e-8451-3beff4f272f4-kube-api-access-nqjm7\" (UniqueName: \"kubernetes.io/projected/5b24cdad-99eb-488e-8451-3beff4f272f4-kube-api-access-nqjm7\") pod \"5b24cdad-99eb-488e-8451-3beff4f272f4\" (UID: \"5b24cdad-99eb-488e-8451-3beff4f272f4\") "
Oct 31 14:17:32.655394 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.655229    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7457057f-33a8-42cc-a099-43796099665f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7457057f-33a8-42cc-a099-43796099665f-plugins-config-volume\") pod \"7457057f-33a8-42cc-a099-43796099665f\" (UID: \"7457057f-33a8-42cc-a099-43796099665f\") "
Oct 31 14:17:32.655560 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.655531    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b24cdad-99eb-488e-8451-3beff4f272f4-plugins-config-volume" pod "5b24cdad-99eb-488e-8451-3beff4f272f4" (UID: "5b24cdad-99eb-488e-8451-3beff4f272f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:17:32.655625 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.655585    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7457057f-33a8-42cc-a099-43796099665f-plugins-config-volume" pod "7457057f-33a8-42cc-a099-43796099665f" (UID: "7457057f-33a8-42cc-a099-43796099665f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:17:32.657539 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.657514    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7457057f-33a8-42cc-a099-43796099665f-kube-api-access-54vdm" pod "7457057f-33a8-42cc-a099-43796099665f" (UID: "7457057f-33a8-42cc-a099-43796099665f"). InnerVolumeSpecName "kube-api-access-54vdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:17:32.657701 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.657681    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b24cdad-99eb-488e-8451-3beff4f272f4-kube-api-access-nqjm7" pod "5b24cdad-99eb-488e-8451-3beff4f272f4" (UID: "5b24cdad-99eb-488e-8451-3beff4f272f4"). InnerVolumeSpecName "kube-api-access-nqjm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:17:32.756350 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.756256    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7457057f-33a8-42cc-a099-43796099665f-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:32.756350 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.756287    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54vdm\" (UniqueName: \"kubernetes.io/projected/7457057f-33a8-42cc-a099-43796099665f-kube-api-access-54vdm\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:32.756350 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.756297    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b24cdad-99eb-488e-8451-3beff4f272f4-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:32.756350 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.756306    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqjm7\" (UniqueName: \"kubernetes.io/projected/5b24cdad-99eb-488e-8451-3beff4f272f4-kube-api-access-nqjm7\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:32.871727 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.871692    4428 generic.go:356] "Generic (PLEG): container finished" podID="7457057f-33a8-42cc-a099-43796099665f" containerID="318e99ab7777e91ba20ba7416c5b9f51a325ce35b41a05979a297444db7f40be" exitCode=0
Oct 31 14:17:32.871973 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.871756    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"
Oct 31 14:17:32.871973 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.871776    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6" event={"ID":"7457057f-33a8-42cc-a099-43796099665f","Type":"ContainerDied","Data":"318e99ab7777e91ba20ba7416c5b9f51a325ce35b41a05979a297444db7f40be"}
Oct 31 14:17:32.871973 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.871845    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6" event={"ID":"7457057f-33a8-42cc-a099-43796099665f","Type":"ContainerDied","Data":"e93c4dcc001a2b1d5e8d2fd71454f07f573432613a72b1e835585b5fec73b105"}
Oct 31 14:17:32.871973 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.871896    4428 scope.go:122] "RemoveContainer" containerID="318e99ab7777e91ba20ba7416c5b9f51a325ce35b41a05979a297444db7f40be"
Oct 31 14:17:32.873383 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.873359    4428 generic.go:356] "Generic (PLEG): container finished" podID="5b24cdad-99eb-488e-8451-3beff4f272f4" containerID="453ff708c1c1539a7cd76f8f936c153f65ecfa5c2d5fd27c54ee2d679176e895" exitCode=0
Oct 31 14:17:32.873525 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.873438    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"
Oct 31 14:17:32.873596 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.873450    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs" event={"ID":"5b24cdad-99eb-488e-8451-3beff4f272f4","Type":"ContainerDied","Data":"453ff708c1c1539a7cd76f8f936c153f65ecfa5c2d5fd27c54ee2d679176e895"}
Oct 31 14:17:32.873664 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.873598    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs" event={"ID":"5b24cdad-99eb-488e-8451-3beff4f272f4","Type":"ContainerDied","Data":"d2467a593cd7fb387745c781d1f2694013d52d55c4214065e1f732b868b4753d"}
Oct 31 14:17:32.882499 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.882479    4428 scope.go:122] "RemoveContainer" containerID="318e99ab7777e91ba20ba7416c5b9f51a325ce35b41a05979a297444db7f40be"
Oct 31 14:17:32.882824 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:17:32.882802    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"318e99ab7777e91ba20ba7416c5b9f51a325ce35b41a05979a297444db7f40be\": container with ID starting with 318e99ab7777e91ba20ba7416c5b9f51a325ce35b41a05979a297444db7f40be not found: ID does not exist" containerID="318e99ab7777e91ba20ba7416c5b9f51a325ce35b41a05979a297444db7f40be"
Oct 31 14:17:32.882957 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.882831    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"318e99ab7777e91ba20ba7416c5b9f51a325ce35b41a05979a297444db7f40be"} err="failed to get container status \"318e99ab7777e91ba20ba7416c5b9f51a325ce35b41a05979a297444db7f40be\": rpc error: code = NotFound desc = could not find container \"318e99ab7777e91ba20ba7416c5b9f51a325ce35b41a05979a297444db7f40be\": container with ID starting with 318e99ab7777e91ba20ba7416c5b9f51a325ce35b41a05979a297444db7f40be not found: ID does not exist"
Oct 31 14:17:32.882957 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.882850    4428 scope.go:122] "RemoveContainer" containerID="453ff708c1c1539a7cd76f8f936c153f65ecfa5c2d5fd27c54ee2d679176e895"
Oct 31 14:17:32.893596 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.893571    4428 scope.go:122] "RemoveContainer" containerID="453ff708c1c1539a7cd76f8f936c153f65ecfa5c2d5fd27c54ee2d679176e895"
Oct 31 14:17:32.893973 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:17:32.893951    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453ff708c1c1539a7cd76f8f936c153f65ecfa5c2d5fd27c54ee2d679176e895\": container with ID starting with 453ff708c1c1539a7cd76f8f936c153f65ecfa5c2d5fd27c54ee2d679176e895 not found: ID does not exist" containerID="453ff708c1c1539a7cd76f8f936c153f65ecfa5c2d5fd27c54ee2d679176e895"
Oct 31 14:17:32.894063 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.893993    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453ff708c1c1539a7cd76f8f936c153f65ecfa5c2d5fd27c54ee2d679176e895"} err="failed to get container status \"453ff708c1c1539a7cd76f8f936c153f65ecfa5c2d5fd27c54ee2d679176e895\": rpc error: code = NotFound desc = could not find container \"453ff708c1c1539a7cd76f8f936c153f65ecfa5c2d5fd27c54ee2d679176e895\": container with ID starting with 453ff708c1c1539a7cd76f8f936c153f65ecfa5c2d5fd27c54ee2d679176e895 not found: ID does not exist"
Oct 31 14:17:32.896278 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.896259    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"]
Oct 31 14:17:32.899655 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.899626    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1008181-77b484dbbc-4vdt6"]
Oct 31 14:17:32.914167 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.914144    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"]
Oct 31 14:17:32.919254 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:32.919229    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1008181-55d6d486cf-wpkvs"]
Oct 31 14:17:33.561680 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:33.561587    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"]
Oct 31 14:17:33.562167 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:33.561830    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz" podUID="bd571eff-4b33-473e-9994-a4b1328a066c" containerName="istio-proxy" containerID="cri-o://b45b9cc70fad4992af69254477c2740818a18006bcecf3ae3ae54ea37cde8061" gracePeriod=30
Oct 31 14:17:33.665135 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:33.665023    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:34.755200 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:34.755117    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b24cdad-99eb-488e-8451-3beff4f272f4" path="/var/lib/kubelet/pods/5b24cdad-99eb-488e-8451-3beff4f272f4/volumes"
Oct 31 14:17:34.755594 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:34.755496    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7457057f-33a8-42cc-a099-43796099665f" path="/var/lib/kubelet/pods/7457057f-33a8-42cc-a099-43796099665f/volumes"
Oct 31 14:17:35.013102 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.013014    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:35.406652 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.406562    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"]
Oct 31 14:17:35.407116 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.407099    4428 cpu_manager.go:414] "containerMap: removing container" podUID="7457057f-33a8-42cc-a099-43796099665f" containerName="payload-pre-processing"
Oct 31 14:17:35.407172 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.407121    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7457057f-33a8-42cc-a099-43796099665f" containerName="payload-pre-processing"
Oct 31 14:17:35.407172 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.407162    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b7ecbf14-9916-42a1-84bf-0945115f7d24" containerName="maas-api"
Oct 31 14:17:35.407172 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.407169    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7ecbf14-9916-42a1-84bf-0945115f7d24" containerName="maas-api"
Oct 31 14:17:35.407287 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.407184    4428 cpu_manager.go:414] "containerMap: removing container" podUID="5b24cdad-99eb-488e-8451-3beff4f272f4" containerName="payload-processing"
Oct 31 14:17:35.407287 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.407192    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b24cdad-99eb-488e-8451-3beff4f272f4" containerName="payload-processing"
Oct 31 14:17:35.407363 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.407329    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7ecbf14-9916-42a1-84bf-0945115f7d24" containerName="maas-api"
Oct 31 14:17:35.407363 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.407346    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b24cdad-99eb-488e-8451-3beff4f272f4" containerName="payload-processing"
Oct 31 14:17:35.407452 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.407361    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="7457057f-33a8-42cc-a099-43796099665f" containerName="payload-pre-processing"
Oct 31 14:17:35.412069 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.412044    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.421841 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.421814    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-c67e06-openshift-default-dockercfg-qjn8c\""
Oct 31 14:17:35.430605 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.430579    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"]
Oct 31 14:17:35.486578 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.486541    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-workload-socket\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.486761 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.486581    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-podinfo\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.486761 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.486611    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdk59\" (UniqueName: \"kubernetes.io/projected/22ff8fc2-3544-442a-9112-58a5e88fa4eb-kube-api-access-pdk59\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.486840 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.486777    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-data\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.486895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.486832    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-envoy\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.486938 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.486892    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-credential-socket\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.486976 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.486942    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-workload-certs\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.487046 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.487028    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istiod-ca-cert\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.487101 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.487087    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-token\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.507036 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.506998    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:35.588379 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.588342    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-envoy\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.588605 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.588391    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-credential-socket\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.588605 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.588432    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-workload-certs\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.588605 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.588473    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istiod-ca-cert\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.588605 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.588527    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-token\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.588605 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.588562    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-workload-socket\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.588605 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.588591    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-podinfo\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.589022 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.588719    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdk59\" (UniqueName: \"kubernetes.io/projected/22ff8fc2-3544-442a-9112-58a5e88fa4eb-kube-api-access-pdk59\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.589022 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.588896    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-credential-socket\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.589022 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.589016    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-data\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.589207 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.589020    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-workload-socket\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.589381 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.589346    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-data\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.589524 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.589501    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-workload-certs\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.589649 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.589498    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istiod-ca-cert\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.591003 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.590978    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-envoy\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.591629 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.591610    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-podinfo\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.602322 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.602291    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-token\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.602629 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.602608    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdk59\" (UniqueName: \"kubernetes.io/projected/22ff8fc2-3544-442a-9112-58a5e88fa4eb-kube-api-access-pdk59\") pod \"e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") " pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:35.659855 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.659754    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:35.728205 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:35.728165    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:36.019793 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:36.019765    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"]
Oct 31 14:17:36.021652 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:17:36.021626    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ff8fc2_3544_442a_9112_58a5e88fa4eb.slice/crio-d961872b6e9a83a3ba52ff17598c90d7011fa2285375cf7b966802572446196d WatchSource:0}: Error finding container d961872b6e9a83a3ba52ff17598c90d7011fa2285375cf7b966802572446196d: Status 404 returned error can't find the container with id d961872b6e9a83a3ba52ff17598c90d7011fa2285375cf7b966802572446196d
Oct 31 14:17:36.024276 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:36.024243    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:17:36.024376 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:36.024321    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:17:36.024376 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:36.024350    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:17:36.297583 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:36.297490    4428 patch_prober.go:29] interesting pod/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 14:17:36.297583 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:36.297557    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz" podUID="bd571eff-4b33-473e-9994-a4b1328a066c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 14:17:36.898536 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:36.898433    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7" event={"ID":"22ff8fc2-3544-442a-9112-58a5e88fa4eb","Type":"ContainerStarted","Data":"47cb7ba1a4e648a74df23952995f1cfc651fbe2b33cecee384bc9093282f2df3"}
Oct 31 14:17:36.898536 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:36.898495    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7" event={"ID":"22ff8fc2-3544-442a-9112-58a5e88fa4eb","Type":"ContainerStarted","Data":"d961872b6e9a83a3ba52ff17598c90d7011fa2285375cf7b966802572446196d"}
Oct 31 14:17:36.926167 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:36.926113    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7" podStartSLOduration=1.92609772 podStartE2EDuration="1.92609772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:17:36.923250988 +0000 UTC m=+4454.321599607" watchObservedRunningTime="2026-10-31 14:17:36.92609772 +0000 UTC m=+4454.324446343"
Oct 31 14:17:37.729340 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:37.729302    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:37.734749 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:37.734715    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:37.904471 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:37.904435    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:37.905573 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:37.905554    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:38.818681 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.818656    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:17:38.910705 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.910619    4428 generic.go:356] "Generic (PLEG): container finished" podID="bd571eff-4b33-473e-9994-a4b1328a066c" containerID="b45b9cc70fad4992af69254477c2740818a18006bcecf3ae3ae54ea37cde8061" exitCode=0
Oct 31 14:17:38.910705 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.910670    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"
Oct 31 14:17:38.911028 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.910696    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz" event={"ID":"bd571eff-4b33-473e-9994-a4b1328a066c","Type":"ContainerDied","Data":"b45b9cc70fad4992af69254477c2740818a18006bcecf3ae3ae54ea37cde8061"}
Oct 31 14:17:38.911028 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.910759    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz" event={"ID":"bd571eff-4b33-473e-9994-a4b1328a066c","Type":"ContainerDied","Data":"6d3793d20c8f97620bd3601d777ac32ba08d1c95a9b0430015ba4aea7e3c3653"}
Oct 31 14:17:38.911028 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.910793    4428 scope.go:122] "RemoveContainer" containerID="b45b9cc70fad4992af69254477c2740818a18006bcecf3ae3ae54ea37cde8061"
Oct 31 14:17:38.920556 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.920538    4428 scope.go:122] "RemoveContainer" containerID="36778f36963aedaf5a88f8fdefe87a47b5229714ebc34db7dcdc9d0f5f625a00"
Oct 31 14:17:38.923784 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.923762    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd571eff-4b33-473e-9994-a4b1328a066c-istio-token\" (UniqueName: \"kubernetes.io/projected/bd571eff-4b33-473e-9994-a4b1328a066c-istio-token\") pod \"bd571eff-4b33-473e-9994-a4b1328a066c\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") "
Oct 31 14:17:38.923899 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.923807    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-workload-certs\") pod \"bd571eff-4b33-473e-9994-a4b1328a066c\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") "
Oct 31 14:17:38.923899 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.923843    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-credential-socket\") pod \"bd571eff-4b33-473e-9994-a4b1328a066c\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") "
Oct 31 14:17:38.923999 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.923900    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd571eff-4b33-473e-9994-a4b1328a066c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd571eff-4b33-473e-9994-a4b1328a066c-istiod-ca-cert\") pod \"bd571eff-4b33-473e-9994-a4b1328a066c\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") "
Oct 31 14:17:38.923999 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.923934    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-workload-socket\") pod \"bd571eff-4b33-473e-9994-a4b1328a066c\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") "
Oct 31 14:17:38.924127 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.924008    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-istio-envoy\") pod \"bd571eff-4b33-473e-9994-a4b1328a066c\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") "
Oct 31 14:17:38.924127 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.924049    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-istio-data\") pod \"bd571eff-4b33-473e-9994-a4b1328a066c\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") "
Oct 31 14:17:38.924127 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.924084    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd571eff-4b33-473e-9994-a4b1328a066c-kube-api-access-pb6f8\" (UniqueName: \"kubernetes.io/projected/bd571eff-4b33-473e-9994-a4b1328a066c-kube-api-access-pb6f8\") pod \"bd571eff-4b33-473e-9994-a4b1328a066c\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") "
Oct 31 14:17:38.924127 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.924104    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-workload-certs" pod "bd571eff-4b33-473e-9994-a4b1328a066c" (UID: "bd571eff-4b33-473e-9994-a4b1328a066c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:17:38.924127 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.924117    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-credential-socket" pod "bd571eff-4b33-473e-9994-a4b1328a066c" (UID: "bd571eff-4b33-473e-9994-a4b1328a066c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:17:38.924401 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.924184    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd571eff-4b33-473e-9994-a4b1328a066c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd571eff-4b33-473e-9994-a4b1328a066c-istio-podinfo\") pod \"bd571eff-4b33-473e-9994-a4b1328a066c\" (UID: \"bd571eff-4b33-473e-9994-a4b1328a066c\") "
Oct 31 14:17:38.924401 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.924245    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-workload-socket" pod "bd571eff-4b33-473e-9994-a4b1328a066c" (UID: "bd571eff-4b33-473e-9994-a4b1328a066c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:17:38.924401 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.924335    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-istio-data" pod "bd571eff-4b33-473e-9994-a4b1328a066c" (UID: "bd571eff-4b33-473e-9994-a4b1328a066c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:17:38.924401 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.924359    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd571eff-4b33-473e-9994-a4b1328a066c-istiod-ca-cert" pod "bd571eff-4b33-473e-9994-a4b1328a066c" (UID: "bd571eff-4b33-473e-9994-a4b1328a066c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:17:38.924605 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.924542    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:38.924605 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.924562    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:38.924605 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.924576    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd571eff-4b33-473e-9994-a4b1328a066c-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:38.924605 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.924589    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:38.925036 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.924605    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:38.926585 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.926562    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-istio-envoy" pod "bd571eff-4b33-473e-9994-a4b1328a066c" (UID: "bd571eff-4b33-473e-9994-a4b1328a066c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:17:38.926708 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.926625    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd571eff-4b33-473e-9994-a4b1328a066c-istio-token" pod "bd571eff-4b33-473e-9994-a4b1328a066c" (UID: "bd571eff-4b33-473e-9994-a4b1328a066c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:17:38.926708 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.926627    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd571eff-4b33-473e-9994-a4b1328a066c-istio-podinfo" pod "bd571eff-4b33-473e-9994-a4b1328a066c" (UID: "bd571eff-4b33-473e-9994-a4b1328a066c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:17:38.926708 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.926664    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd571eff-4b33-473e-9994-a4b1328a066c-kube-api-access-pb6f8" pod "bd571eff-4b33-473e-9994-a4b1328a066c" (UID: "bd571eff-4b33-473e-9994-a4b1328a066c"). InnerVolumeSpecName "kube-api-access-pb6f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:17:38.930310 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.930291    4428 scope.go:122] "RemoveContainer" containerID="b45b9cc70fad4992af69254477c2740818a18006bcecf3ae3ae54ea37cde8061"
Oct 31 14:17:38.930595 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:17:38.930576    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45b9cc70fad4992af69254477c2740818a18006bcecf3ae3ae54ea37cde8061\": container with ID starting with b45b9cc70fad4992af69254477c2740818a18006bcecf3ae3ae54ea37cde8061 not found: ID does not exist" containerID="b45b9cc70fad4992af69254477c2740818a18006bcecf3ae3ae54ea37cde8061"
Oct 31 14:17:38.930669 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.930604    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45b9cc70fad4992af69254477c2740818a18006bcecf3ae3ae54ea37cde8061"} err="failed to get container status \"b45b9cc70fad4992af69254477c2740818a18006bcecf3ae3ae54ea37cde8061\": rpc error: code = NotFound desc = could not find container \"b45b9cc70fad4992af69254477c2740818a18006bcecf3ae3ae54ea37cde8061\": container with ID starting with b45b9cc70fad4992af69254477c2740818a18006bcecf3ae3ae54ea37cde8061 not found: ID does not exist"
Oct 31 14:17:38.930669 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.930623    4428 scope.go:122] "RemoveContainer" containerID="36778f36963aedaf5a88f8fdefe87a47b5229714ebc34db7dcdc9d0f5f625a00"
Oct 31 14:17:38.930857 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:17:38.930836    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36778f36963aedaf5a88f8fdefe87a47b5229714ebc34db7dcdc9d0f5f625a00\": container with ID starting with 36778f36963aedaf5a88f8fdefe87a47b5229714ebc34db7dcdc9d0f5f625a00 not found: ID does not exist" containerID="36778f36963aedaf5a88f8fdefe87a47b5229714ebc34db7dcdc9d0f5f625a00"
Oct 31 14:17:38.931082 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:38.930933    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36778f36963aedaf5a88f8fdefe87a47b5229714ebc34db7dcdc9d0f5f625a00"} err="failed to get container status \"36778f36963aedaf5a88f8fdefe87a47b5229714ebc34db7dcdc9d0f5f625a00\": rpc error: code = NotFound desc = could not find container \"36778f36963aedaf5a88f8fdefe87a47b5229714ebc34db7dcdc9d0f5f625a00\": container with ID starting with 36778f36963aedaf5a88f8fdefe87a47b5229714ebc34db7dcdc9d0f5f625a00 not found: ID does not exist"
Oct 31 14:17:39.025729 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:39.025682    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd571eff-4b33-473e-9994-a4b1328a066c-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:39.025729 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:39.025724    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd571eff-4b33-473e-9994-a4b1328a066c-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:39.026002 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:39.025738    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd571eff-4b33-473e-9994-a4b1328a066c-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:39.026002 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:39.025755    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb6f8\" (UniqueName: \"kubernetes.io/projected/bd571eff-4b33-473e-9994-a4b1328a066c-kube-api-access-pb6f8\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:17:39.197246 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:39.197128    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:39.238593 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:39.238558    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"]
Oct 31 14:17:39.246278 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:39.246241    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a1008181-openshift-default-57476b49cc-b4htz"]
Oct 31 14:17:39.353652 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:39.353602    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:40.323765 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.323726    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"]
Oct 31 14:17:40.324432 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.324409    4428 cpu_manager.go:414] "containerMap: removing container" podUID="bd571eff-4b33-473e-9994-a4b1328a066c" containerName="istio-proxy"
Oct 31 14:17:40.324478 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.324436    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd571eff-4b33-473e-9994-a4b1328a066c" containerName="istio-proxy"
Oct 31 14:17:40.324514 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.324480    4428 cpu_manager.go:414] "containerMap: removing container" podUID="bd571eff-4b33-473e-9994-a4b1328a066c" containerName="istio-proxy"
Oct 31 14:17:40.324514 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.324487    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd571eff-4b33-473e-9994-a4b1328a066c" containerName="istio-proxy"
Oct 31 14:17:40.324595 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.324577    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd571eff-4b33-473e-9994-a4b1328a066c" containerName="istio-proxy"
Oct 31 14:17:40.324632 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.324597    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd571eff-4b33-473e-9994-a4b1328a066c" containerName="istio-proxy"
Oct 31 14:17:40.329741 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.329717    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:17:40.339058 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.338316    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-c67e06-serving-cert\""
Oct 31 14:17:40.340146 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.340116    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"]
Oct 31 14:17:40.421925 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.421891    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:40.441601 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.441567    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b327326b-f621-4a40-85ef-777a60dff4ce-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r\" (UID: \"b327326b-f621-4a40-85ef-777a60dff4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:17:40.441786 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.441628    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b327326b-f621-4a40-85ef-777a60dff4ce-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r\" (UID: \"b327326b-f621-4a40-85ef-777a60dff4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:17:40.441786 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.441758    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b327326b-f621-4a40-85ef-777a60dff4ce-metrics-certs\") pod \"maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r\" (UID: \"b327326b-f621-4a40-85ef-777a60dff4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:17:40.441897 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.441786    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvhmq\" (UniqueName: \"kubernetes.io/projected/b327326b-f621-4a40-85ef-777a60dff4ce-kube-api-access-gvhmq\") pod \"maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r\" (UID: \"b327326b-f621-4a40-85ef-777a60dff4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:17:40.452131 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.452082    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"]
Oct 31 14:17:40.456946 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.456898    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"
Oct 31 14:17:40.460906 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.460845    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"]
Oct 31 14:17:40.461535 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.461504    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-c67e06-dockercfg-k87fn\""
Oct 31 14:17:40.462706 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.462682    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-c67e06\""
Oct 31 14:17:40.497322 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.497288    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"]
Oct 31 14:17:40.500952 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.500933    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"
Oct 31 14:17:40.513786 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.513758    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"]
Oct 31 14:17:40.542982 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.542947    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72933bbe-b1f9-4a34-8568-be0dddd2b5a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8\" (UID: \"72933bbe-b1f9-4a34-8568-be0dddd2b5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"
Oct 31 14:17:40.543143 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.542999    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b327326b-f621-4a40-85ef-777a60dff4ce-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r\" (UID: \"b327326b-f621-4a40-85ef-777a60dff4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:17:40.543143 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.543079    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9wm2\" (UniqueName: \"kubernetes.io/projected/72933bbe-b1f9-4a34-8568-be0dddd2b5a4-kube-api-access-c9wm2\") pod \"payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8\" (UID: \"72933bbe-b1f9-4a34-8568-be0dddd2b5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"
Oct 31 14:17:40.543291 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:17:40.543162    4428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-serving-cert: secret "maas-api-e2e-authcfg-http-c67e06-serving-cert" not found
Oct 31 14:17:40.543291 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.543209    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b327326b-f621-4a40-85ef-777a60dff4ce-metrics-certs\") pod \"maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r\" (UID: \"b327326b-f621-4a40-85ef-777a60dff4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:17:40.543291 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:17:40.543222    4428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b327326b-f621-4a40-85ef-777a60dff4ce-maas-api-tls podName:b327326b-f621-4a40-85ef-777a60dff4ce nodeName:}" failed. No retries permitted until 2026-10-31 14:17:41.043208212 +0000 UTC m=+4458.441556842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b327326b-f621-4a40-85ef-777a60dff4ce-maas-api-tls") pod "maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r" (UID: "b327326b-f621-4a40-85ef-777a60dff4ce") : secret "maas-api-e2e-authcfg-http-c67e06-serving-cert" not found
Oct 31 14:17:40.543425 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.543347    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhmq\" (UniqueName: \"kubernetes.io/projected/b327326b-f621-4a40-85ef-777a60dff4ce-kube-api-access-gvhmq\") pod \"maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r\" (UID: \"b327326b-f621-4a40-85ef-777a60dff4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:17:40.543425 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.543413    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b327326b-f621-4a40-85ef-777a60dff4ce-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r\" (UID: \"b327326b-f621-4a40-85ef-777a60dff4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:17:40.544026 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.544002    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b327326b-f621-4a40-85ef-777a60dff4ce-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r\" (UID: \"b327326b-f621-4a40-85ef-777a60dff4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:17:40.546124 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.546100    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b327326b-f621-4a40-85ef-777a60dff4ce-metrics-certs\") pod \"maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r\" (UID: \"b327326b-f621-4a40-85ef-777a60dff4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:17:40.559761 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.559727    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhmq\" (UniqueName: \"kubernetes.io/projected/b327326b-f621-4a40-85ef-777a60dff4ce-kube-api-access-gvhmq\") pod \"maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r\" (UID: \"b327326b-f621-4a40-85ef-777a60dff4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:17:40.591981 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.587495    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:40.644957 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.644910    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72933bbe-b1f9-4a34-8568-be0dddd2b5a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8\" (UID: \"72933bbe-b1f9-4a34-8568-be0dddd2b5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"
Oct 31 14:17:40.645158 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.645008    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9wm2\" (UniqueName: \"kubernetes.io/projected/72933bbe-b1f9-4a34-8568-be0dddd2b5a4-kube-api-access-c9wm2\") pod \"payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8\" (UID: \"72933bbe-b1f9-4a34-8568-be0dddd2b5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"
Oct 31 14:17:40.645158 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.645078    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894f26ab-0672-47af-8d07-cb699150dc6f-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj\" (UID: \"894f26ab-0672-47af-8d07-cb699150dc6f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"
Oct 31 14:17:40.645270 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.645186    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzd6q\" (UniqueName: \"kubernetes.io/projected/894f26ab-0672-47af-8d07-cb699150dc6f-kube-api-access-zzd6q\") pod \"payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj\" (UID: \"894f26ab-0672-47af-8d07-cb699150dc6f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"
Oct 31 14:17:40.645638 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.645610    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72933bbe-b1f9-4a34-8568-be0dddd2b5a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8\" (UID: \"72933bbe-b1f9-4a34-8568-be0dddd2b5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"
Oct 31 14:17:40.658350 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.658318    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9wm2\" (UniqueName: \"kubernetes.io/projected/72933bbe-b1f9-4a34-8568-be0dddd2b5a4-kube-api-access-c9wm2\") pod \"payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8\" (UID: \"72933bbe-b1f9-4a34-8568-be0dddd2b5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"
Oct 31 14:17:40.746367 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.746331    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzd6q\" (UniqueName: \"kubernetes.io/projected/894f26ab-0672-47af-8d07-cb699150dc6f-kube-api-access-zzd6q\") pod \"payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj\" (UID: \"894f26ab-0672-47af-8d07-cb699150dc6f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"
Oct 31 14:17:40.746543 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.746453    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894f26ab-0672-47af-8d07-cb699150dc6f-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj\" (UID: \"894f26ab-0672-47af-8d07-cb699150dc6f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"
Oct 31 14:17:40.747114 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.747094    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894f26ab-0672-47af-8d07-cb699150dc6f-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj\" (UID: \"894f26ab-0672-47af-8d07-cb699150dc6f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"
Oct 31 14:17:40.754099 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.754075    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:40.754397 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.754376    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd571eff-4b33-473e-9994-a4b1328a066c" path="/var/lib/kubelet/pods/bd571eff-4b33-473e-9994-a4b1328a066c/volumes"
Oct 31 14:17:40.760533 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.760510    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzd6q\" (UniqueName: \"kubernetes.io/projected/894f26ab-0672-47af-8d07-cb699150dc6f-kube-api-access-zzd6q\") pod \"payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj\" (UID: \"894f26ab-0672-47af-8d07-cb699150dc6f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"
Oct 31 14:17:40.773221 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.773197    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"
Oct 31 14:17:40.816323 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:40.815939    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"
Oct 31 14:17:41.049544 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:41.049512    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b327326b-f621-4a40-85ef-777a60dff4ce-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r\" (UID: \"b327326b-f621-4a40-85ef-777a60dff4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:17:41.052261 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:41.052235    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b327326b-f621-4a40-85ef-777a60dff4ce-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r\" (UID: \"b327326b-f621-4a40-85ef-777a60dff4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:17:41.056759 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:41.056732    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"]
Oct 31 14:17:41.058238 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:17:41.058207    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72933bbe_b1f9_4a34_8568_be0dddd2b5a4.slice/crio-bd04a677c395985b0bc2e06a17e33f86c897468f3b64b477c9962016b1f28a5e WatchSource:0}: Error finding container bd04a677c395985b0bc2e06a17e33f86c897468f3b64b477c9962016b1f28a5e: Status 404 returned error can't find the container with id bd04a677c395985b0bc2e06a17e33f86c897468f3b64b477c9962016b1f28a5e
Oct 31 14:17:41.112117 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:41.112076    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"]
Oct 31 14:17:41.113467 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:17:41.113437    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894f26ab_0672_47af_8d07_cb699150dc6f.slice/crio-f8744873d2cc7a9e5913c60d6e125662d52da763e514634a85ae3188df761798 WatchSource:0}: Error finding container f8744873d2cc7a9e5913c60d6e125662d52da763e514634a85ae3188df761798: Status 404 returned error can't find the container with id f8744873d2cc7a9e5913c60d6e125662d52da763e514634a85ae3188df761798
Oct 31 14:17:41.248058 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:41.248023    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:17:41.492226 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:41.492192    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:41.543174 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:41.543146    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"]
Oct 31 14:17:41.543930 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:17:41.543903    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb327326b_f621_4a40_85ef_777a60dff4ce.slice/crio-097a824ebce71717fdb04ec6e0aff4c94cebc4e933cd212b943168d1a3602096 WatchSource:0}: Error finding container 097a824ebce71717fdb04ec6e0aff4c94cebc4e933cd212b943168d1a3602096: Status 404 returned error can't find the container with id 097a824ebce71717fdb04ec6e0aff4c94cebc4e933cd212b943168d1a3602096
Oct 31 14:17:41.929818 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:41.929708    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj" event={"ID":"894f26ab-0672-47af-8d07-cb699150dc6f","Type":"ContainerStarted","Data":"21a90d151d933eebb380172b9d673dfeb0fdb0a70735c90699192785974d94da"}
Oct 31 14:17:41.929818 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:41.929784    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"
Oct 31 14:17:41.929818 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:41.929813    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj" event={"ID":"894f26ab-0672-47af-8d07-cb699150dc6f","Type":"ContainerStarted","Data":"f8744873d2cc7a9e5913c60d6e125662d52da763e514634a85ae3188df761798"}
Oct 31 14:17:41.931178 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:41.931145    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r" event={"ID":"b327326b-f621-4a40-85ef-777a60dff4ce","Type":"ContainerStarted","Data":"097a824ebce71717fdb04ec6e0aff4c94cebc4e933cd212b943168d1a3602096"}
Oct 31 14:17:41.932771 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:41.932728    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8" event={"ID":"72933bbe-b1f9-4a34-8568-be0dddd2b5a4","Type":"ContainerStarted","Data":"0a88ade0c6f441e8df13e38132501d9feaa68486b76f0cc7072aca4fe9fd819c"}
Oct 31 14:17:41.933040 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:41.932798    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8" event={"ID":"72933bbe-b1f9-4a34-8568-be0dddd2b5a4","Type":"ContainerStarted","Data":"bd04a677c395985b0bc2e06a17e33f86c897468f3b64b477c9962016b1f28a5e"}
Oct 31 14:17:41.933040 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:41.932905    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"
Oct 31 14:17:41.960405 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:41.960345    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj" podStartSLOduration=1.9603307650000001 podStartE2EDuration="1.960330765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:17:41.958354883 +0000 UTC m=+4459.356703509" watchObservedRunningTime="2026-10-31 14:17:41.960330765 +0000 UTC m=+4459.358679423"
Oct 31 14:17:41.982056 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:41.981634    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8" podStartSLOduration=1.981614467 podStartE2EDuration="1.981614467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:17:41.981118253 +0000 UTC m=+4459.379466869" watchObservedRunningTime="2026-10-31 14:17:41.981614467 +0000 UTC m=+4459.379963094"
Oct 31 14:17:42.403760 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:42.403715    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:42.942987 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:42.942947    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r" event={"ID":"b327326b-f621-4a40-85ef-777a60dff4ce","Type":"ContainerStarted","Data":"78054ad41b314e92126c78ddc839cbac17aacbe349a888f7baafb5d9f0e134f8"}
Oct 31 14:17:42.968256 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:42.968201    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r" podStartSLOduration=2.531485082 podStartE2EDuration="2.968185021s" totalImagesPullingTime="436.699939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:17:42.967148065 +0000 UTC m=+4460.365496689" watchObservedRunningTime="2026-10-31 14:17:42.968185021 +0000 UTC m=+4460.366533642"
Oct 31 14:17:43.138024 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:43.137991    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:43.948923 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:43.948883    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:17:44.033494 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:44.033464    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:44.741714 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:44.741677    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:45.525801 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:45.525763    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:46.337936 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:46.337896    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:47.156183 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:47.156143    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:47.934350 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:47.934312    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:48.745482 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:17:48.745443    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:17:48.771437 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:48.771398    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:49.631558 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:49.631507    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:49.967263 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:49.967235    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:17:50.439102 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:50.439003    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:51.312061 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:51.312026    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:52.135717 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:52.135613    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:52.925397 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:52.925358    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:52.947043 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:52.946621    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"
Oct 31 14:17:52.947043 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:52.946746    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"
Oct 31 14:17:53.065234 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:53.065052    4428 patch_prober.go:29] interesting pod/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.100:15021/healthz/ready\": read tcp 10.134.0.2:50470->10.134.1.100:15021: read: connection reset by peer" start-of-body=
Oct 31 14:17:53.065234 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:53.065126    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7" podUID="22ff8fc2-3544-442a-9112-58a5e88fa4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.100:15021/healthz/ready\": read tcp 10.134.0.2:50470->10.134.1.100:15021: read: connection reset by peer"
Oct 31 14:17:53.802409 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:53.802374    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:54.031473 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:54.031439    4428 generic.go:356] "Generic (PLEG): container finished" podID="22ff8fc2-3544-442a-9112-58a5e88fa4eb" containerID="47cb7ba1a4e648a74df23952995f1cfc651fbe2b33cecee384bc9093282f2df3" exitCode=0
Oct 31 14:17:54.031926 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:54.031508    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7" event={"ID":"22ff8fc2-3544-442a-9112-58a5e88fa4eb","Type":"ContainerDied","Data":"47cb7ba1a4e648a74df23952995f1cfc651fbe2b33cecee384bc9093282f2df3"}
Oct 31 14:17:54.031992 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:54.031944    4428 scope.go:122] "RemoveContainer" containerID="47cb7ba1a4e648a74df23952995f1cfc651fbe2b33cecee384bc9093282f2df3"
Oct 31 14:17:54.033196 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:54.033165    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:17:54.033322 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:54.033261    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:17:54.033322 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:54.033315    4428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 14:17:54.210897 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:54.210822    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:54.543328 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:54.543292    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:55.039732 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:55.039697    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7" event={"ID":"22ff8fc2-3544-442a-9112-58a5e88fa4eb","Type":"ContainerStarted","Data":"1ab8e9a734671cdba2718935b7d218110c4bb589b8a9603aaa3a1abf3a115699"}
Oct 31 14:17:55.322033 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:55.321923    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:55.729279 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:55.729249    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:17:55.729486 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:55.729438    4428 patch_prober.go:29] interesting pod/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused" start-of-body=
Oct 31 14:17:55.729551 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:55.729484    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7" podUID="22ff8fc2-3544-442a-9112-58a5e88fa4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused"
Oct 31 14:17:56.074987 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:56.072377    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:56.728582 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:56.728548    4428 patch_prober.go:29] interesting pod/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused" start-of-body=
Oct 31 14:17:56.728786 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:56.728600    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7" podUID="22ff8fc2-3544-442a-9112-58a5e88fa4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused"
Oct 31 14:17:56.941284 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:56.941246    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:57.729075 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:57.729036    4428 patch_prober.go:29] interesting pod/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused" start-of-body=
Oct 31 14:17:57.729594 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:57.729566    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7" podUID="22ff8fc2-3544-442a-9112-58a5e88fa4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused"
Oct 31 14:17:57.736018 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:57.735989    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:58.545788 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:58.545752    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:58.729393 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:58.729360    4428 patch_prober.go:29] interesting pod/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused" start-of-body=
Oct 31 14:17:58.729828 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:58.729414    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7" podUID="22ff8fc2-3544-442a-9112-58a5e88fa4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused"
Oct 31 14:17:59.335731 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:59.335697    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:17:59.728884 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:59.728841    4428 patch_prober.go:29] interesting pod/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused" start-of-body=
Oct 31 14:17:59.729073 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:17:59.728933    4428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7" podUID="22ff8fc2-3544-442a-9112-58a5e88fa4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused"
Oct 31 14:18:00.080565 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:00.080452    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:00.733297 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:00.733255    4428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:18:00.733696 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:00.733676    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:18:00.734361 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:00.734343    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:18:00.946049 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:00.946008    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:01.675816 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:01.675786    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:02.445259 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:02.445225    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:03.286523 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:03.286491    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:03.977233 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:03.977185    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:04.772334 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:04.772297    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:05.485191 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:05.485146    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:06.236857 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:06.236779    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:07.123379 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:07.123344    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:07.951030 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:07.950995    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:08.839052 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:08.839011    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:09.631256 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:09.631225    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:10.403632 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:10.403593    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:11.141428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:11.141391    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:11.941897 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:11.941837    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:12.738946 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:12.738901    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:13.548081 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:13.548051    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:14.338213 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:14.338183    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:15.142621 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:15.142585    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:15.923951 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:15.923920    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:16.648454 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:16.648419    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:16.815446 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:16.815410    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:17.379749 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:17.379712    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:17.537207 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:17.537172    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:18.139312 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:18.139280    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:18.740239 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:18:18.740199    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:18:19.049464 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:19.049369    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:19.835216 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:19.835168    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:20.634839 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:20.634806    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:21.349367 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:21.349333    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:22.137139 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:22.137102    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:22.927156 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:22.927122    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:23.733695 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:23.733654    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:24.588834 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:24.588781    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:25.278850 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:25.278814    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:25.943432 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:25.943402    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:26.706428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:26.706398    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:27.439616 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:27.439587    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:28.224570 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:28.224537    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:29.137698 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:29.137664    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:29.744023 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:29.743990    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:30.544532 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:30.544499    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:31.326968 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:31.326926    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:32.044784 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:32.044747    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:32.833811 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:32.833778    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:33.029127 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:33.029087    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"]
Oct 31 14:18:33.032886 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:33.032841    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nn2gt"]
Oct 31 14:18:33.547846 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:33.547807    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:34.254378 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:34.254342    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:34.755464 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:34.755432    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaa58b67-cc9e-4608-a17c-330435693cb5" path="/var/lib/kubelet/pods/aaa58b67-cc9e-4608-a17c-330435693cb5/volumes"
Oct 31 14:18:35.043773 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:35.043688    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:35.939077 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:35.939043    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:36.554561 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:36.554525    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:36.568669 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:36.568645    4428 scope.go:122] "RemoveContainer" containerID="479f9eb32b4578887ef9a5fdc053b80ccd248b69f8e1fa7a8b0377f7db6a95a5"
Oct 31 14:18:36.740339 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:36.740298    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:37.320576 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:37.320542    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:38.134480 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:38.134448    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:38.964038 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:38.964006    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:39.732549 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:39.732519    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:40.528836 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:40.528799    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:41.328816 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:41.328775    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:42.407733 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:42.407700    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:42.935251 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:42.935218    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:43.739347 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:43.739316    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:44.543037 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:44.543004    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:45.423193 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:45.423150    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:46.290921 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:46.290841    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:46.943325 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:46.943294    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:47.733258 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:47.733224    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:48.466366 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:48.466326    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:48.740797 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:18:48.740712    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:18:49.230161 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:49.230104    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:50.037361 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:50.037328    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:50.843528 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:50.843445    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:51.636143 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:51.636106    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:52.419041 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:52.419008    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:53.154912 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:53.154856    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:53.936011 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:53.935974    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:54.027528 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:54.027493    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"]
Oct 31 14:18:54.030789 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:54.030762    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-b45f9"]
Oct 31 14:18:54.754409 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:54.754371    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c45f885f-cf08-4803-b7e1-6a16d279ff8a" path="/var/lib/kubelet/pods/c45f885f-cf08-4803-b7e1-6a16d279ff8a/volumes"
Oct 31 14:18:54.853990 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:54.853959    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:55.614531 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:55.614499    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:56.440151 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:56.440116    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:57.131580 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:57.131544    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:57.932085 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:57.932050    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:58.638914 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:58.638855    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:18:59.429623 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:18:59.429591    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:00.237908 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:00.237857    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:00.940974 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:00.940925    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:01.674463 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:01.674426    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:02.439800 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:02.439763    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:03.218149 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:03.218111    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:04.026920 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:04.026833    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:04.742847 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:04.742807    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:05.447945 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:05.447914    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:06.261322 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:06.261287    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:07.047158 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:07.047118    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:07.739940 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:07.739905    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:08.532307 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:08.532271    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:09.252572 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:09.252541    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:10.037987 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:10.037953    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:10.835413 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:10.835356    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:11.539187 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:11.539158    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:12.424207 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:12.424165    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:13.137501 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:13.137459    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:13.915553 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:13.915517    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:14.911383 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:14.911335    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:15.734306 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:15.734271    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:16.554616 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:16.554551    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:16.833392 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:16.833258    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:17.239492 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:17.239453    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:17.462925 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:17.462856    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:18.050487 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:18.050438    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:18.741209 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:19:18.741171    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:19:18.828376 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:18.828344    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:19.633010 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:19.632977    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:20.348626 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:20.348593    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:21.137436 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:21.137406    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:21.936160 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:21.936120    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:22.740073 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:22.740016    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:23.526749 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:23.526681    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:24.254178 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:24.254127    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:24.308252 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:24.308222    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"]
Oct 31 14:19:24.331134 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:24.331099    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"]
Oct 31 14:19:24.443084 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:24.443038    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:24.620350 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:24.620226    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:25.044751 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:25.044711    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:29.308513 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.308467    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj" podUID="894f26ab-0672-47af-8d07-cb699150dc6f" containerName="payload-processing" containerID="cri-o://21a90d151d933eebb380172b9d673dfeb0fdb0a70735c90699192785974d94da" gracePeriod=25
Oct 31 14:19:29.331628 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.331600    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8" podUID="72933bbe-b1f9-4a34-8568-be0dddd2b5a4" containerName="payload-pre-processing" containerID="cri-o://0a88ade0c6f441e8df13e38132501d9feaa68486b76f0cc7072aca4fe9fd819c" gracePeriod=25
Oct 31 14:19:29.545029 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.544990    4428 generic.go:356] "Generic (PLEG): container finished" podID="72933bbe-b1f9-4a34-8568-be0dddd2b5a4" containerID="0a88ade0c6f441e8df13e38132501d9feaa68486b76f0cc7072aca4fe9fd819c" exitCode=0
Oct 31 14:19:29.545198 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.545080    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8" event={"ID":"72933bbe-b1f9-4a34-8568-be0dddd2b5a4","Type":"ContainerDied","Data":"0a88ade0c6f441e8df13e38132501d9feaa68486b76f0cc7072aca4fe9fd819c"}
Oct 31 14:19:29.547285 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.547249    4428 generic.go:356] "Generic (PLEG): container finished" podID="894f26ab-0672-47af-8d07-cb699150dc6f" containerID="21a90d151d933eebb380172b9d673dfeb0fdb0a70735c90699192785974d94da" exitCode=0
Oct 31 14:19:29.547434 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.547294    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj" event={"ID":"894f26ab-0672-47af-8d07-cb699150dc6f","Type":"ContainerDied","Data":"21a90d151d933eebb380172b9d673dfeb0fdb0a70735c90699192785974d94da"}
Oct 31 14:19:29.584222 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.584195    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"
Oct 31 14:19:29.620874 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.620841    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"
Oct 31 14:19:29.685977 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.685940    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72933bbe-b1f9-4a34-8568-be0dddd2b5a4-kube-api-access-c9wm2\" (UniqueName: \"kubernetes.io/projected/72933bbe-b1f9-4a34-8568-be0dddd2b5a4-kube-api-access-c9wm2\") pod \"72933bbe-b1f9-4a34-8568-be0dddd2b5a4\" (UID: \"72933bbe-b1f9-4a34-8568-be0dddd2b5a4\") "
Oct 31 14:19:29.686192 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.685998    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/894f26ab-0672-47af-8d07-cb699150dc6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894f26ab-0672-47af-8d07-cb699150dc6f-plugins-config-volume\") pod \"894f26ab-0672-47af-8d07-cb699150dc6f\" (UID: \"894f26ab-0672-47af-8d07-cb699150dc6f\") "
Oct 31 14:19:29.686192 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.686027    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/894f26ab-0672-47af-8d07-cb699150dc6f-kube-api-access-zzd6q\" (UniqueName: \"kubernetes.io/projected/894f26ab-0672-47af-8d07-cb699150dc6f-kube-api-access-zzd6q\") pod \"894f26ab-0672-47af-8d07-cb699150dc6f\" (UID: \"894f26ab-0672-47af-8d07-cb699150dc6f\") "
Oct 31 14:19:29.686192 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.686083    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72933bbe-b1f9-4a34-8568-be0dddd2b5a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72933bbe-b1f9-4a34-8568-be0dddd2b5a4-plugins-config-volume\") pod \"72933bbe-b1f9-4a34-8568-be0dddd2b5a4\" (UID: \"72933bbe-b1f9-4a34-8568-be0dddd2b5a4\") "
Oct 31 14:19:29.686443 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.686418    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894f26ab-0672-47af-8d07-cb699150dc6f-plugins-config-volume" pod "894f26ab-0672-47af-8d07-cb699150dc6f" (UID: "894f26ab-0672-47af-8d07-cb699150dc6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:19:29.686501 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.686475    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72933bbe-b1f9-4a34-8568-be0dddd2b5a4-plugins-config-volume" pod "72933bbe-b1f9-4a34-8568-be0dddd2b5a4" (UID: "72933bbe-b1f9-4a34-8568-be0dddd2b5a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:19:29.688284 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.688260    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72933bbe-b1f9-4a34-8568-be0dddd2b5a4-kube-api-access-c9wm2" pod "72933bbe-b1f9-4a34-8568-be0dddd2b5a4" (UID: "72933bbe-b1f9-4a34-8568-be0dddd2b5a4"). InnerVolumeSpecName "kube-api-access-c9wm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:19:29.688355 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.688337    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894f26ab-0672-47af-8d07-cb699150dc6f-kube-api-access-zzd6q" pod "894f26ab-0672-47af-8d07-cb699150dc6f" (UID: "894f26ab-0672-47af-8d07-cb699150dc6f"). InnerVolumeSpecName "kube-api-access-zzd6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:19:29.787534 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.787421    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9wm2\" (UniqueName: \"kubernetes.io/projected/72933bbe-b1f9-4a34-8568-be0dddd2b5a4-kube-api-access-c9wm2\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:29.787534 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.787457    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894f26ab-0672-47af-8d07-cb699150dc6f-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:29.787534 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.787476    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzd6q\" (UniqueName: \"kubernetes.io/projected/894f26ab-0672-47af-8d07-cb699150dc6f-kube-api-access-zzd6q\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:29.787534 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:29.787492    4428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72933bbe-b1f9-4a34-8568-be0dddd2b5a4-plugins-config-volume\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:30.557852 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.555759    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"
Oct 31 14:19:30.557852 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.555768    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8" event={"ID":"72933bbe-b1f9-4a34-8568-be0dddd2b5a4","Type":"ContainerDied","Data":"bd04a677c395985b0bc2e06a17e33f86c897468f3b64b477c9962016b1f28a5e"}
Oct 31 14:19:30.557852 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.556309    4428 scope.go:122] "RemoveContainer" containerID="0a88ade0c6f441e8df13e38132501d9feaa68486b76f0cc7072aca4fe9fd819c"
Oct 31 14:19:30.560673 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.558266    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj" event={"ID":"894f26ab-0672-47af-8d07-cb699150dc6f","Type":"ContainerDied","Data":"f8744873d2cc7a9e5913c60d6e125662d52da763e514634a85ae3188df761798"}
Oct 31 14:19:30.560673 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.558552    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"
Oct 31 14:19:30.571794 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.571752    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"]
Oct 31 14:19:30.572107 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.572034    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r" podUID="b327326b-f621-4a40-85ef-777a60dff4ce" containerName="maas-api" containerID="cri-o://78054ad41b314e92126c78ddc839cbac17aacbe349a888f7baafb5d9f0e134f8" gracePeriod=30
Oct 31 14:19:30.572193 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.572111    4428 scope.go:122] "RemoveContainer" containerID="21a90d151d933eebb380172b9d673dfeb0fdb0a70735c90699192785974d94da"
Oct 31 14:19:30.621590 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.621547    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"]
Oct 31 14:19:30.628824 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.628785    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-c67e06-bcf9db5df-584lj"]
Oct 31 14:19:30.642614 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.642581    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"]
Oct 31 14:19:30.650004 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.649536    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-c67e06-687bdc754c-q7sm8"]
Oct 31 14:19:30.674590 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.674548    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:30.754676 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.754643    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72933bbe-b1f9-4a34-8568-be0dddd2b5a4" path="/var/lib/kubelet/pods/72933bbe-b1f9-4a34-8568-be0dddd2b5a4/volumes"
Oct 31 14:19:30.755353 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.755225    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="894f26ab-0672-47af-8d07-cb699150dc6f" path="/var/lib/kubelet/pods/894f26ab-0672-47af-8d07-cb699150dc6f/volumes"
Oct 31 14:19:30.836570 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.836537    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:30.849198 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.849165    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:19:30.896129 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.896086    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b327326b-f621-4a40-85ef-777a60dff4ce-kube-api-access-gvhmq\" (UniqueName: \"kubernetes.io/projected/b327326b-f621-4a40-85ef-777a60dff4ce-kube-api-access-gvhmq\") pod \"b327326b-f621-4a40-85ef-777a60dff4ce\" (UID: \"b327326b-f621-4a40-85ef-777a60dff4ce\") "
Oct 31 14:19:30.896346 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.896182    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b327326b-f621-4a40-85ef-777a60dff4ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b327326b-f621-4a40-85ef-777a60dff4ce-metrics-certs\") pod \"b327326b-f621-4a40-85ef-777a60dff4ce\" (UID: \"b327326b-f621-4a40-85ef-777a60dff4ce\") "
Oct 31 14:19:30.896346 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.896256    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b327326b-f621-4a40-85ef-777a60dff4ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b327326b-f621-4a40-85ef-777a60dff4ce-maas-api-tls\") pod \"b327326b-f621-4a40-85ef-777a60dff4ce\" (UID: \"b327326b-f621-4a40-85ef-777a60dff4ce\") "
Oct 31 14:19:30.896495 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.896365    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b327326b-f621-4a40-85ef-777a60dff4ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b327326b-f621-4a40-85ef-777a60dff4ce-openshift-service-ca\") pod \"b327326b-f621-4a40-85ef-777a60dff4ce\" (UID: \"b327326b-f621-4a40-85ef-777a60dff4ce\") "
Oct 31 14:19:30.897082 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.897009    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b327326b-f621-4a40-85ef-777a60dff4ce-openshift-service-ca" pod "b327326b-f621-4a40-85ef-777a60dff4ce" (UID: "b327326b-f621-4a40-85ef-777a60dff4ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:19:30.898625 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.898595    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b327326b-f621-4a40-85ef-777a60dff4ce-maas-api-tls" pod "b327326b-f621-4a40-85ef-777a60dff4ce" (UID: "b327326b-f621-4a40-85ef-777a60dff4ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:19:30.899122 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.899103    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b327326b-f621-4a40-85ef-777a60dff4ce-metrics-certs" pod "b327326b-f621-4a40-85ef-777a60dff4ce" (UID: "b327326b-f621-4a40-85ef-777a60dff4ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:19:30.900254 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.900235    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b327326b-f621-4a40-85ef-777a60dff4ce-kube-api-access-gvhmq" pod "b327326b-f621-4a40-85ef-777a60dff4ce" (UID: "b327326b-f621-4a40-85ef-777a60dff4ce"). InnerVolumeSpecName "kube-api-access-gvhmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:19:30.989631 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.989554    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:30.997925 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.997899    4428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b327326b-f621-4a40-85ef-777a60dff4ce-openshift-service-ca\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:30.997925 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.997924    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvhmq\" (UniqueName: \"kubernetes.io/projected/b327326b-f621-4a40-85ef-777a60dff4ce-kube-api-access-gvhmq\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:30.998142 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.997935    4428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b327326b-f621-4a40-85ef-777a60dff4ce-metrics-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:30.998142 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:30.997945    4428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b327326b-f621-4a40-85ef-777a60dff4ce-maas-api-tls\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:31.408307 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:31.408272    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:31.565587 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:31.565548    4428 generic.go:356] "Generic (PLEG): container finished" podID="b327326b-f621-4a40-85ef-777a60dff4ce" containerID="78054ad41b314e92126c78ddc839cbac17aacbe349a888f7baafb5d9f0e134f8" exitCode=0
Oct 31 14:19:31.565587 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:31.565592    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"
Oct 31 14:19:31.566138 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:31.565626    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r" event={"ID":"b327326b-f621-4a40-85ef-777a60dff4ce","Type":"ContainerDied","Data":"78054ad41b314e92126c78ddc839cbac17aacbe349a888f7baafb5d9f0e134f8"}
Oct 31 14:19:31.566138 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:31.565678    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r" event={"ID":"b327326b-f621-4a40-85ef-777a60dff4ce","Type":"ContainerDied","Data":"097a824ebce71717fdb04ec6e0aff4c94cebc4e933cd212b943168d1a3602096"}
Oct 31 14:19:31.566138 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:31.565702    4428 scope.go:122] "RemoveContainer" containerID="78054ad41b314e92126c78ddc839cbac17aacbe349a888f7baafb5d9f0e134f8"
Oct 31 14:19:31.576540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:31.576515    4428 scope.go:122] "RemoveContainer" containerID="78054ad41b314e92126c78ddc839cbac17aacbe349a888f7baafb5d9f0e134f8"
Oct 31 14:19:31.576900 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:19:31.576854    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78054ad41b314e92126c78ddc839cbac17aacbe349a888f7baafb5d9f0e134f8\": container with ID starting with 78054ad41b314e92126c78ddc839cbac17aacbe349a888f7baafb5d9f0e134f8 not found: ID does not exist" containerID="78054ad41b314e92126c78ddc839cbac17aacbe349a888f7baafb5d9f0e134f8"
Oct 31 14:19:31.576977 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:31.576921    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78054ad41b314e92126c78ddc839cbac17aacbe349a888f7baafb5d9f0e134f8"} err="failed to get container status \"78054ad41b314e92126c78ddc839cbac17aacbe349a888f7baafb5d9f0e134f8\": rpc error: code = NotFound desc = could not find container \"78054ad41b314e92126c78ddc839cbac17aacbe349a888f7baafb5d9f0e134f8\": container with ID starting with 78054ad41b314e92126c78ddc839cbac17aacbe349a888f7baafb5d9f0e134f8 not found: ID does not exist"
Oct 31 14:19:31.590083 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:31.590054    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"]
Oct 31 14:19:31.594584 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:31.594560    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-c67e06-57bccc9b9f-c272r"]
Oct 31 14:19:32.755862 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:32.755819    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b327326b-f621-4a40-85ef-777a60dff4ce" path="/var/lib/kubelet/pods/b327326b-f621-4a40-85ef-777a60dff4ce/volumes"
Oct 31 14:19:36.594678 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:36.594650    4428 scope.go:122] "RemoveContainer" containerID="a138dde338fbd1e66d5dd1c0729d60b0e664641920244f193f1e15de9f67ec90"
Oct 31 14:19:37.729779 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:37.729746    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:37.886788 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:37.886757    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:38.008149 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:38.008047    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"]
Oct 31 14:19:38.008344 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:38.008321    4428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7" podUID="22ff8fc2-3544-442a-9112-58a5e88fa4eb" containerName="istio-proxy" containerID="cri-o://1ab8e9a734671cdba2718935b7d218110c4bb589b8a9603aaa3a1abf3a115699" gracePeriod=30
Oct 31 14:19:38.108323 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:38.108282    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:19:43.300258 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.300229    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:19:43.345332 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.345291    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-podinfo\") pod \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") "
Oct 31 14:19:43.345540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.345368    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ff8fc2-3544-442a-9112-58a5e88fa4eb-kube-api-access-pdk59\" (UniqueName: \"kubernetes.io/projected/22ff8fc2-3544-442a-9112-58a5e88fa4eb-kube-api-access-pdk59\") pod \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") "
Oct 31 14:19:43.345540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.345398    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-workload-certs\") pod \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") "
Oct 31 14:19:43.345540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.345428    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-data\") pod \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") "
Oct 31 14:19:43.345540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.345471    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istiod-ca-cert\") pod \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") "
Oct 31 14:19:43.345540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.345523    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-envoy\") pod \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") "
Oct 31 14:19:43.345835 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.345550    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-workload-socket\") pod \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") "
Oct 31 14:19:43.345835 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.345590    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-token\" (UniqueName: \"kubernetes.io/projected/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-token\") pod \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") "
Oct 31 14:19:43.345835 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.345676    4428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-credential-socket\") pod \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\" (UID: \"22ff8fc2-3544-442a-9112-58a5e88fa4eb\") "
Oct 31 14:19:43.345835 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.345738    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-workload-certs" pod "22ff8fc2-3544-442a-9112-58a5e88fa4eb" (UID: "22ff8fc2-3544-442a-9112-58a5e88fa4eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:19:43.345835 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.345773    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-data" pod "22ff8fc2-3544-442a-9112-58a5e88fa4eb" (UID: "22ff8fc2-3544-442a-9112-58a5e88fa4eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:19:43.346125 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.346045    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-workload-socket" pod "22ff8fc2-3544-442a-9112-58a5e88fa4eb" (UID: "22ff8fc2-3544-442a-9112-58a5e88fa4eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:19:43.346125 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.346056    4428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-workload-certs\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:43.346125 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.346077    4428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-data\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:43.346290 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.346134    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istiod-ca-cert" pod "22ff8fc2-3544-442a-9112-58a5e88fa4eb" (UID: "22ff8fc2-3544-442a-9112-58a5e88fa4eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:19:43.346376 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.346351    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-credential-socket" pod "22ff8fc2-3544-442a-9112-58a5e88fa4eb" (UID: "22ff8fc2-3544-442a-9112-58a5e88fa4eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:19:43.348443 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.348416    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-token" pod "22ff8fc2-3544-442a-9112-58a5e88fa4eb" (UID: "22ff8fc2-3544-442a-9112-58a5e88fa4eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:19:43.348443 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.348432    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-envoy" pod "22ff8fc2-3544-442a-9112-58a5e88fa4eb" (UID: "22ff8fc2-3544-442a-9112-58a5e88fa4eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:19:43.348607 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.348447    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ff8fc2-3544-442a-9112-58a5e88fa4eb-kube-api-access-pdk59" pod "22ff8fc2-3544-442a-9112-58a5e88fa4eb" (UID: "22ff8fc2-3544-442a-9112-58a5e88fa4eb"). InnerVolumeSpecName "kube-api-access-pdk59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:19:43.348847 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.348815    4428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-podinfo" pod "22ff8fc2-3544-442a-9112-58a5e88fa4eb" (UID: "22ff8fc2-3544-442a-9112-58a5e88fa4eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 14:19:43.446798 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.446757    4428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-token\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:43.446798 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.446791    4428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-credential-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:43.446798 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.446804    4428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-podinfo\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:43.447096 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.446820    4428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdk59\" (UniqueName: \"kubernetes.io/projected/22ff8fc2-3544-442a-9112-58a5e88fa4eb-kube-api-access-pdk59\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:43.447096 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.446837    4428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istiod-ca-cert\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:43.447096 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.446851    4428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-istio-envoy\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:43.447096 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.446862    4428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ff8fc2-3544-442a-9112-58a5e88fa4eb-workload-socket\") on node \"ip-10-0-136-168.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 14:19:43.654066 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.653973    4428 generic.go:356] "Generic (PLEG): container finished" podID="22ff8fc2-3544-442a-9112-58a5e88fa4eb" containerID="1ab8e9a734671cdba2718935b7d218110c4bb589b8a9603aaa3a1abf3a115699" exitCode=0
Oct 31 14:19:43.654066 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.654049    4428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"
Oct 31 14:19:43.654066 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.654054    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7" event={"ID":"22ff8fc2-3544-442a-9112-58a5e88fa4eb","Type":"ContainerDied","Data":"1ab8e9a734671cdba2718935b7d218110c4bb589b8a9603aaa3a1abf3a115699"}
Oct 31 14:19:43.654350 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.654118    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7" event={"ID":"22ff8fc2-3544-442a-9112-58a5e88fa4eb","Type":"ContainerDied","Data":"d961872b6e9a83a3ba52ff17598c90d7011fa2285375cf7b966802572446196d"}
Oct 31 14:19:43.654350 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.654142    4428 scope.go:122] "RemoveContainer" containerID="1ab8e9a734671cdba2718935b7d218110c4bb589b8a9603aaa3a1abf3a115699"
Oct 31 14:19:43.664600 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.664579    4428 scope.go:122] "RemoveContainer" containerID="47cb7ba1a4e648a74df23952995f1cfc651fbe2b33cecee384bc9093282f2df3"
Oct 31 14:19:43.673616 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.673592    4428 scope.go:122] "RemoveContainer" containerID="1ab8e9a734671cdba2718935b7d218110c4bb589b8a9603aaa3a1abf3a115699"
Oct 31 14:19:43.673952 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:19:43.673932    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab8e9a734671cdba2718935b7d218110c4bb589b8a9603aaa3a1abf3a115699\": container with ID starting with 1ab8e9a734671cdba2718935b7d218110c4bb589b8a9603aaa3a1abf3a115699 not found: ID does not exist" containerID="1ab8e9a734671cdba2718935b7d218110c4bb589b8a9603aaa3a1abf3a115699"
Oct 31 14:19:43.674018 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.673961    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab8e9a734671cdba2718935b7d218110c4bb589b8a9603aaa3a1abf3a115699"} err="failed to get container status \"1ab8e9a734671cdba2718935b7d218110c4bb589b8a9603aaa3a1abf3a115699\": rpc error: code = NotFound desc = could not find container \"1ab8e9a734671cdba2718935b7d218110c4bb589b8a9603aaa3a1abf3a115699\": container with ID starting with 1ab8e9a734671cdba2718935b7d218110c4bb589b8a9603aaa3a1abf3a115699 not found: ID does not exist"
Oct 31 14:19:43.674018 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.673978    4428 scope.go:122] "RemoveContainer" containerID="47cb7ba1a4e648a74df23952995f1cfc651fbe2b33cecee384bc9093282f2df3"
Oct 31 14:19:43.674287 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:19:43.674267    4428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47cb7ba1a4e648a74df23952995f1cfc651fbe2b33cecee384bc9093282f2df3\": container with ID starting with 47cb7ba1a4e648a74df23952995f1cfc651fbe2b33cecee384bc9093282f2df3 not found: ID does not exist" containerID="47cb7ba1a4e648a74df23952995f1cfc651fbe2b33cecee384bc9093282f2df3"
Oct 31 14:19:43.674347 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.674296    4428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47cb7ba1a4e648a74df23952995f1cfc651fbe2b33cecee384bc9093282f2df3"} err="failed to get container status \"47cb7ba1a4e648a74df23952995f1cfc651fbe2b33cecee384bc9093282f2df3\": rpc error: code = NotFound desc = could not find container \"47cb7ba1a4e648a74df23952995f1cfc651fbe2b33cecee384bc9093282f2df3\": container with ID starting with 47cb7ba1a4e648a74df23952995f1cfc651fbe2b33cecee384bc9093282f2df3 not found: ID does not exist"
Oct 31 14:19:43.688917 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.683789    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"]
Oct 31 14:19:43.695044 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:43.695016    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-c67e06-openshift-default-565cd9b9d9-mnjj7"]
Oct 31 14:19:44.754016 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:19:44.753981    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ff8fc2-3544-442a-9112-58a5e88fa4eb" path="/var/lib/kubelet/pods/22ff8fc2-3544-442a-9112-58a5e88fa4eb/volumes"
Oct 31 14:19:48.741766 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:19:48.741727    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:20:05.078012 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:05.077969    4428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-66bvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 14:20:05.078833 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:05.078792    4428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-66bvk" podUID="ca1c7578-b050-429c-8a3d-6d771125649c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 14:20:16.796961 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:16.796925    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:20:18.747743 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:20:18.747703    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:20:19.196582 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:19.196495    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:20:29.826595 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:29.826543    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:20:36.638924 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:36.638822    4428 scope.go:122] "RemoveContainer" containerID="b05495d789219ab0131dba9bf011c93707c7aadc46016111d33cf74418b3efd0"
Oct 31 14:20:38.584419 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:38.584384    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58ff7d8b4-skjvb_30763b38-29e0-4998-843f-5471f4f621db/manager/0.log"
Oct 31 14:20:38.792932 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:38.792902    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-t88kb_276f7225-018f-4d4d-90a1-3ed0086cafd0/manager/0.log"
Oct 31 14:20:39.714855 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:39.714825    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6brb8_874303da-8bf8-44e8-bdb9-d9aadf7dd956/server/0.log"
Oct 31 14:20:39.902020 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:39.901994    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-f5jn6_b027618a-47db-416a-9081-13db8719d2d6/manager/0.log"
Oct 31 14:20:40.132030 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:40.131949    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vkln9_25cbee37-655e-42f3-a41c-36ff91af938e/proxy/0.log"
Oct 31 14:20:40.935154 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:40.935120    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-64fbfb966c-6jndx_1c4554c2-52c9-4417-9388-cff7dc8add81/maas-api/0.log"
Oct 31 14:20:41.496065 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:41.496037    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2ec5dc37-xbjjf_d500889f-a602-4b38-b66c-db0f5500c483/revoke-keys/0.log"
Oct 31 14:20:41.667945 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:41.667909    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-db7l6_64524ae5-3f14-4ed5-9b9a-a3187b168224/revoke-keys/0.log"
Oct 31 14:20:41.856990 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:41.856887    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-kqmz2_20d5d735-f0a7-4756-96c1-7fbfa8018803/revoke-keys/0.log"
Oct 31 14:20:42.397782 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:42.397740    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-h7fm4_90cd394c-0002-4feb-aa67-2ddc2bfc9411/postgres/0.log"
Oct 31 14:20:44.714105 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:44.714070    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cpbtr_4ada56cc-ea77-4a96-bb9b-3e26162a1aa6/manager/0.log"
Oct 31 14:20:46.296301 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:46.296268    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7485898565-4fxqz_10d7c0b8-2c3c-4ead-938b-0062041d31ae/kube-auth-proxy/0.log"
Oct 31 14:20:47.033072 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:47.033041    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-66bvk_ca1c7578-b050-429c-8a3d-6d771125649c/istio-proxy/0.log"
Oct 31 14:20:47.390533 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:47.390446    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-7sz9s_aa86ed88-56ab-4fc4-b6a4-bc7732a8db25/istio-proxy/0.log"
Oct 31 14:20:47.944013 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:47.943983    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-n56bm_aaa34a48-48ac-4d05-8d9d-0d0c0a870f88/istio-proxy/0.log"
Oct 31 14:20:48.683572 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:48.683542    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-km7m9_db129029-9467-4431-bc52-40c0524ff214/payload-pre-processing/0.log"
Oct 31 14:20:48.740621 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:20:48.740581    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:20:48.852338 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:48.852312    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w9wjk_e328741d-31fd-40db-8cbb-8c9b518f4cde/payload-processing/0.log"
Oct 31 14:20:50.033626 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:50.033597    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg_5c86b906-a276-4e67-8e58-79a17118f5fd/main/0.log"
Oct 31 14:20:50.215072 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:50.215038    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-596gr_c20e3218-9e4a-43a1-b46e-050de8eb25c4/main/0.log"
Oct 31 14:20:50.573755 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:50.573725    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-hql47_ea978050-20b0-4148-ab41-75844fe1b637/main/0.log"
Oct 31 14:20:50.754428 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:50.754385    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59_cd0ac048-d186-4f3c-929d-eb870afbc801/main/0.log"
Oct 31 14:20:50.936059 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:20:50.935978    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8b7s6_149a9b52-ef95-4569-b947-e6e14d707339/main/0.log"
Oct 31 14:21:01.374489 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:01.374456    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58ff7d8b4-skjvb_30763b38-29e0-4998-843f-5471f4f621db/manager/0.log"
Oct 31 14:21:01.584231 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:01.584201    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-t88kb_276f7225-018f-4d4d-90a1-3ed0086cafd0/manager/0.log"
Oct 31 14:21:02.417040 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:02.417011    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6brb8_874303da-8bf8-44e8-bdb9-d9aadf7dd956/server/0.log"
Oct 31 14:21:02.585947 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:02.585917    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-f5jn6_b027618a-47db-416a-9081-13db8719d2d6/manager/0.log"
Oct 31 14:21:02.784264 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:02.784232    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vkln9_25cbee37-655e-42f3-a41c-36ff91af938e/proxy/0.log"
Oct 31 14:21:03.326806 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:03.326764    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-64fbfb966c-6jndx_1c4554c2-52c9-4417-9388-cff7dc8add81/maas-api/0.log"
Oct 31 14:21:03.890520 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:03.890478    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2ec5dc37-xbjjf_d500889f-a602-4b38-b66c-db0f5500c483/revoke-keys/0.log"
Oct 31 14:21:04.032345 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:04.032309    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"]
Oct 31 14:21:04.035407 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:04.035378    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kqmz2"]
Oct 31 14:21:04.077032 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:04.077004    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-db7l6_64524ae5-3f14-4ed5-9b9a-a3187b168224/revoke-keys/0.log"
Oct 31 14:21:04.754253 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:04.754222    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d5d735-f0a7-4756-96c1-7fbfa8018803" path="/var/lib/kubelet/pods/20d5d735-f0a7-4756-96c1-7fbfa8018803/volumes"
Oct 31 14:21:04.764508 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:04.764479    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-h7fm4_90cd394c-0002-4feb-aa67-2ddc2bfc9411/postgres/0.log"
Oct 31 14:21:06.672396 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:06.672366    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cpbtr_4ada56cc-ea77-4a96-bb9b-3e26162a1aa6/manager/0.log"
Oct 31 14:21:07.966187 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:07.966159    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7485898565-4fxqz_10d7c0b8-2c3c-4ead-938b-0062041d31ae/kube-auth-proxy/0.log"
Oct 31 14:21:08.782176 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:08.782146    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-66bvk_ca1c7578-b050-429c-8a3d-6d771125649c/istio-proxy/0.log"
Oct 31 14:21:09.151720 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:09.151629    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-7sz9s_aa86ed88-56ab-4fc4-b6a4-bc7732a8db25/istio-proxy/0.log"
Oct 31 14:21:09.732766 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:09.732733    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-n56bm_aaa34a48-48ac-4d05-8d9d-0d0c0a870f88/istio-proxy/0.log"
Oct 31 14:21:10.488552 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:10.488518    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-km7m9_db129029-9467-4431-bc52-40c0524ff214/payload-pre-processing/0.log"
Oct 31 14:21:10.669217 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:10.669185    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w9wjk_e328741d-31fd-40db-8cbb-8c9b518f4cde/payload-processing/0.log"
Oct 31 14:21:11.376886 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:11.376841    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-msvrg_5c86b906-a276-4e67-8e58-79a17118f5fd/main/0.log"
Oct 31 14:21:11.568023 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:11.567987    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-596gr_c20e3218-9e4a-43a1-b46e-050de8eb25c4/main/0.log"
Oct 31 14:21:11.944371 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:11.944337    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-hql47_ea978050-20b0-4148-ab41-75844fe1b637/main/0.log"
Oct 31 14:21:12.126300 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:12.126265    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87cf59_cd0ac048-d186-4f3c-929d-eb870afbc801/main/0.log"
Oct 31 14:21:12.305762 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:12.305725    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8b7s6_149a9b52-ef95-4569-b947-e6e14d707339/main/0.log"
Oct 31 14:21:16.851316 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:16.851277    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:21:17.561017 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:17.560979    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:21:18.741301 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:21:18.741257    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:21:21.026598 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:21.026559    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"]
Oct 31 14:21:21.031251 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:21.031220    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-db7l6"]
Oct 31 14:21:22.754431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:22.754396    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64524ae5-3f14-4ed5-9b9a-a3187b168224" path="/var/lib/kubelet/pods/64524ae5-3f14-4ed5-9b9a-a3187b168224/volumes"
Oct 31 14:21:42.027380 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:42.027334    4428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"]
Oct 31 14:21:42.030182 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:42.030149    4428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ec5dc37-xbjjf"]
Oct 31 14:21:42.753695 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:42.753658    4428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d500889f-a602-4b38-b66c-db0f5500c483" path="/var/lib/kubelet/pods/d500889f-a602-4b38-b66c-db0f5500c483/volumes"
Oct 31 14:21:48.740643 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:21:48.740594    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:21:50.933436 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:50.933406    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w9wjk_e328741d-31fd-40db-8cbb-8c9b518f4cde/payload-processing/0.log"
Oct 31 14:21:51.098126 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:21:51.098095    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w9wjk_e328741d-31fd-40db-8cbb-8c9b518f4cde/payload-processing/0.log"
Oct 31 14:22:16.903164 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:16.903079    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:22:17.613971 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:17.613934    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:22:18.741716 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:22:18.741676    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:22:18.943144 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:18.943110    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:22:19.099431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:19.099339    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:22:36.671161 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:36.671127    4428 scope.go:122] "RemoveContainer" containerID="e70e63e4fad4ab17e6ee49aea4262509cfb1a712fa7ba556cb1bd4d6d1b7a8b8"
Oct 31 14:22:36.679924 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:36.679905    4428 scope.go:122] "RemoveContainer" containerID="d2d9071be184af98632640cd54e19def3726d4d599d625ecc9caf63c4a49b13b"
Oct 31 14:22:37.506442 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:37.506408    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w9wjk_e328741d-31fd-40db-8cbb-8c9b518f4cde/payload-processing/0.log"
Oct 31 14:22:37.697836 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:37.697806    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w9wjk_e328741d-31fd-40db-8cbb-8c9b518f4cde/payload-processing/0.log"
Oct 31 14:22:38.077529 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.077496    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bkxgk/must-gather-7kxsp"]
Oct 31 14:22:38.077972 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.077957    4428 cpu_manager.go:414] "containerMap: removing container" podUID="72933bbe-b1f9-4a34-8568-be0dddd2b5a4" containerName="payload-pre-processing"
Oct 31 14:22:38.078033 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.077974    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72933bbe-b1f9-4a34-8568-be0dddd2b5a4" containerName="payload-pre-processing"
Oct 31 14:22:38.078033 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.077986    4428 cpu_manager.go:414] "containerMap: removing container" podUID="b327326b-f621-4a40-85ef-777a60dff4ce" containerName="maas-api"
Oct 31 14:22:38.078033 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.077991    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b327326b-f621-4a40-85ef-777a60dff4ce" containerName="maas-api"
Oct 31 14:22:38.078033 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.078002    4428 cpu_manager.go:414] "containerMap: removing container" podUID="22ff8fc2-3544-442a-9112-58a5e88fa4eb" containerName="istio-proxy"
Oct 31 14:22:38.078033 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.078007    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ff8fc2-3544-442a-9112-58a5e88fa4eb" containerName="istio-proxy"
Oct 31 14:22:38.078033 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.078023    4428 cpu_manager.go:414] "containerMap: removing container" podUID="22ff8fc2-3544-442a-9112-58a5e88fa4eb" containerName="istio-proxy"
Oct 31 14:22:38.078033 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.078028    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ff8fc2-3544-442a-9112-58a5e88fa4eb" containerName="istio-proxy"
Oct 31 14:22:38.078286 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.078041    4428 cpu_manager.go:414] "containerMap: removing container" podUID="894f26ab-0672-47af-8d07-cb699150dc6f" containerName="payload-processing"
Oct 31 14:22:38.078286 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.078046    4428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="894f26ab-0672-47af-8d07-cb699150dc6f" containerName="payload-processing"
Oct 31 14:22:38.078286 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.078098    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b327326b-f621-4a40-85ef-777a60dff4ce" containerName="maas-api"
Oct 31 14:22:38.078286 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.078107    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ff8fc2-3544-442a-9112-58a5e88fa4eb" containerName="istio-proxy"
Oct 31 14:22:38.078286 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.078115    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ff8fc2-3544-442a-9112-58a5e88fa4eb" containerName="istio-proxy"
Oct 31 14:22:38.078286 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.078121    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="894f26ab-0672-47af-8d07-cb699150dc6f" containerName="payload-processing"
Oct 31 14:22:38.078286 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.078128    4428 memory_manager.go:365] "RemoveStaleState removing state" podUID="72933bbe-b1f9-4a34-8568-be0dddd2b5a4" containerName="payload-pre-processing"
Oct 31 14:22:38.082960 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.082942    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bkxgk/must-gather-7kxsp"
Oct 31 14:22:38.088013 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.087985    4428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bkxgk\"/\"default-dockercfg-smppd\""
Oct 31 14:22:38.088214 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.088196    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bkxgk\"/\"kube-root-ca.crt\""
Oct 31 14:22:38.088289 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.088238    4428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bkxgk\"/\"openshift-service-ca.crt\""
Oct 31 14:22:38.108983 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.108948    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bkxgk/must-gather-7kxsp"]
Oct 31 14:22:38.144525 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.144488    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:22:38.172166 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.172122    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7c5b5ea5-4741-4a15-981f-d70074431a73-must-gather-output\") pod \"must-gather-7kxsp\" (UID: \"7c5b5ea5-4741-4a15-981f-d70074431a73\") " pod="openshift-must-gather-bkxgk/must-gather-7kxsp"
Oct 31 14:22:38.172370 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.172267    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25sbk\" (UniqueName: \"kubernetes.io/projected/7c5b5ea5-4741-4a15-981f-d70074431a73-kube-api-access-25sbk\") pod \"must-gather-7kxsp\" (UID: \"7c5b5ea5-4741-4a15-981f-d70074431a73\") " pod="openshift-must-gather-bkxgk/must-gather-7kxsp"
Oct 31 14:22:38.273424 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.273381    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25sbk\" (UniqueName: \"kubernetes.io/projected/7c5b5ea5-4741-4a15-981f-d70074431a73-kube-api-access-25sbk\") pod \"must-gather-7kxsp\" (UID: \"7c5b5ea5-4741-4a15-981f-d70074431a73\") " pod="openshift-must-gather-bkxgk/must-gather-7kxsp"
Oct 31 14:22:38.273633 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.273499    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7c5b5ea5-4741-4a15-981f-d70074431a73-must-gather-output\") pod \"must-gather-7kxsp\" (UID: \"7c5b5ea5-4741-4a15-981f-d70074431a73\") " pod="openshift-must-gather-bkxgk/must-gather-7kxsp"
Oct 31 14:22:38.273913 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.273861    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7c5b5ea5-4741-4a15-981f-d70074431a73-must-gather-output\") pod \"must-gather-7kxsp\" (UID: \"7c5b5ea5-4741-4a15-981f-d70074431a73\") " pod="openshift-must-gather-bkxgk/must-gather-7kxsp"
Oct 31 14:22:38.292247 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.292215    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25sbk\" (UniqueName: \"kubernetes.io/projected/7c5b5ea5-4741-4a15-981f-d70074431a73-kube-api-access-25sbk\") pod \"must-gather-7kxsp\" (UID: \"7c5b5ea5-4741-4a15-981f-d70074431a73\") " pod="openshift-must-gather-bkxgk/must-gather-7kxsp"
Oct 31 14:22:38.317034 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.317001    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:22:38.404889 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.404788    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bkxgk/must-gather-7kxsp"
Oct 31 14:22:38.694852 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.694795    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bkxgk/must-gather-7kxsp"]
Oct 31 14:22:38.697955 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:22:38.697926    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c5b5ea5_4741_4a15_981f_d70074431a73.slice/crio-3bd8ffb28ae0123b35735e95b37610c135779f5ee0ca2bc32733c3a8276a8eb8 WatchSource:0}: Error finding container 3bd8ffb28ae0123b35735e95b37610c135779f5ee0ca2bc32733c3a8276a8eb8: Status 404 returned error can't find the container with id 3bd8ffb28ae0123b35735e95b37610c135779f5ee0ca2bc32733c3a8276a8eb8
Oct 31 14:22:38.700302 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:38.700282    4428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 14:22:39.561341 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:39.561308    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bkxgk/must-gather-7kxsp" event={"ID":"7c5b5ea5-4741-4a15-981f-d70074431a73","Type":"ContainerStarted","Data":"3bd8ffb28ae0123b35735e95b37610c135779f5ee0ca2bc32733c3a8276a8eb8"}
Oct 31 14:22:40.570356 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:40.570311    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bkxgk/must-gather-7kxsp" event={"ID":"7c5b5ea5-4741-4a15-981f-d70074431a73","Type":"ContainerStarted","Data":"d861fcb2f5d8ec9601dd538e35ea5317d167f683095b43f86fdceec498ef1c55"}
Oct 31 14:22:40.570926 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:40.570373    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bkxgk/must-gather-7kxsp" event={"ID":"7c5b5ea5-4741-4a15-981f-d70074431a73","Type":"ContainerStarted","Data":"9f5177bfae18f0b096c63e7133eb1a62e7bafefd732e16f8f84d9c51b3d89cb4"}
Oct 31 14:22:40.592390 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:40.592327    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bkxgk/must-gather-7kxsp" podStartSLOduration=1.792726666 podStartE2EDuration="2.592306379s" totalImagesPullingTime="799.579713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:22:40.590473478 +0000 UTC m=+4757.988822104" watchObservedRunningTime="2026-10-31 14:22:40.592306379 +0000 UTC m=+4757.990654996"
Oct 31 14:22:41.406584 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:41.406549    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6tcxq_d7a89a36-092a-4d38-8a50-60619cc50832/global-pull-secret-syncer/0.log"
Oct 31 14:22:41.672773 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:41.672681    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log"
Oct 31 14:22:41.879072 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:41.879028    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-168.us-east-2.compute.internal_62681df534845faea6eef1fcf975d228/haproxy/0.log"
Oct 31 14:22:41.890182 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:41.890149    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:22:42.055390 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:42.055351    4428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ff7kw_c1b19e39-5b5d-46f7-b005-a1e5165fe135/konnectivity-agent/0.log" limit=1024
Oct 31 14:22:48.798891 ip-10-0-136-168 kubenswrapper[4428]: E1031 14:22:48.797080    4428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2kwh" containerName="extractor"
Oct 31 14:22:50.035001 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:50.034920    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cpbtr_4ada56cc-ea77-4a96-bb9b-3e26162a1aa6/manager/0.log"
Oct 31 14:22:51.434265 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.434222    4428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"]
Oct 31 14:22:51.444546 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.444357    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:51.453352 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.453315    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"]
Oct 31 14:22:51.476254 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.476215    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-lrsp7_b717bd42-13ab-4398-b858-80dc2aa8b887/kube-state-metrics/0.log"
Oct 31 14:22:51.500842 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.499708    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-lrsp7_b717bd42-13ab-4398-b858-80dc2aa8b887/kube-rbac-proxy-main/0.log"
Oct 31 14:22:51.540351 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.540319    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-lrsp7_b717bd42-13ab-4398-b858-80dc2aa8b887/kube-rbac-proxy-self/0.log"
Oct 31 14:22:51.544113 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.544029    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h24pk\" (UniqueName: \"kubernetes.io/projected/3e65b8e1-fa7d-4bef-bb1b-6e65ad268075-kube-api-access-h24pk\") pod \"perf-node-gather-daemonset-59nhr\" (UID: \"3e65b8e1-fa7d-4bef-bb1b-6e65ad268075\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:51.544445 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.544424    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e65b8e1-fa7d-4bef-bb1b-6e65ad268075-sys\") pod \"perf-node-gather-daemonset-59nhr\" (UID: \"3e65b8e1-fa7d-4bef-bb1b-6e65ad268075\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:51.544710 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.544674    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e65b8e1-fa7d-4bef-bb1b-6e65ad268075-podres\") pod \"perf-node-gather-daemonset-59nhr\" (UID: \"3e65b8e1-fa7d-4bef-bb1b-6e65ad268075\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:51.544924 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.544906    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e65b8e1-fa7d-4bef-bb1b-6e65ad268075-lib-modules\") pod \"perf-node-gather-daemonset-59nhr\" (UID: \"3e65b8e1-fa7d-4bef-bb1b-6e65ad268075\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:51.546814 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.546795    4428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e65b8e1-fa7d-4bef-bb1b-6e65ad268075-proc\") pod \"perf-node-gather-daemonset-59nhr\" (UID: \"3e65b8e1-fa7d-4bef-bb1b-6e65ad268075\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:51.566415 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.566385    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-84cd8cdd9d-wh5xd_b6592e99-9caf-418f-ac38-9f58bc8f672a/metrics-server/0.log"
Oct 31 14:22:51.627838 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.627804    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bsdgs_108cc57c-42a8-43d1-bfbc-2a34de82ba31/node-exporter/0.log"
Oct 31 14:22:51.647502 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.647471    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e65b8e1-fa7d-4bef-bb1b-6e65ad268075-sys\") pod \"perf-node-gather-daemonset-59nhr\" (UID: \"3e65b8e1-fa7d-4bef-bb1b-6e65ad268075\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:51.647502 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.647508    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e65b8e1-fa7d-4bef-bb1b-6e65ad268075-podres\") pod \"perf-node-gather-daemonset-59nhr\" (UID: \"3e65b8e1-fa7d-4bef-bb1b-6e65ad268075\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:51.647763 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.647538    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e65b8e1-fa7d-4bef-bb1b-6e65ad268075-lib-modules\") pod \"perf-node-gather-daemonset-59nhr\" (UID: \"3e65b8e1-fa7d-4bef-bb1b-6e65ad268075\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:51.647763 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.647604    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e65b8e1-fa7d-4bef-bb1b-6e65ad268075-podres\") pod \"perf-node-gather-daemonset-59nhr\" (UID: \"3e65b8e1-fa7d-4bef-bb1b-6e65ad268075\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:51.647763 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.647610    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e65b8e1-fa7d-4bef-bb1b-6e65ad268075-proc\") pod \"perf-node-gather-daemonset-59nhr\" (UID: \"3e65b8e1-fa7d-4bef-bb1b-6e65ad268075\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:51.647763 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.647609    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e65b8e1-fa7d-4bef-bb1b-6e65ad268075-sys\") pod \"perf-node-gather-daemonset-59nhr\" (UID: \"3e65b8e1-fa7d-4bef-bb1b-6e65ad268075\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:51.647763 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.647675    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e65b8e1-fa7d-4bef-bb1b-6e65ad268075-lib-modules\") pod \"perf-node-gather-daemonset-59nhr\" (UID: \"3e65b8e1-fa7d-4bef-bb1b-6e65ad268075\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:51.647763 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.647708    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e65b8e1-fa7d-4bef-bb1b-6e65ad268075-proc\") pod \"perf-node-gather-daemonset-59nhr\" (UID: \"3e65b8e1-fa7d-4bef-bb1b-6e65ad268075\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:51.648123 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.647828    4428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h24pk\" (UniqueName: \"kubernetes.io/projected/3e65b8e1-fa7d-4bef-bb1b-6e65ad268075-kube-api-access-h24pk\") pod \"perf-node-gather-daemonset-59nhr\" (UID: \"3e65b8e1-fa7d-4bef-bb1b-6e65ad268075\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:51.654004 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.653977    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bsdgs_108cc57c-42a8-43d1-bfbc-2a34de82ba31/kube-rbac-proxy/0.log"
Oct 31 14:22:51.663241 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.663209    4428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h24pk\" (UniqueName: \"kubernetes.io/projected/3e65b8e1-fa7d-4bef-bb1b-6e65ad268075-kube-api-access-h24pk\") pod \"perf-node-gather-daemonset-59nhr\" (UID: \"3e65b8e1-fa7d-4bef-bb1b-6e65ad268075\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:51.681536 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.681513    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bsdgs_108cc57c-42a8-43d1-bfbc-2a34de82ba31/init-textfile/0.log"
Oct 31 14:22:51.772850 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:51.772816    4428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:52.061652 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:52.061555    4428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"]
Oct 31 14:22:52.066059 ip-10-0-136-168 kubenswrapper[4428]: W1031 14:22:52.066025    4428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3e65b8e1_fa7d_4bef_bb1b_6e65ad268075.slice/crio-32d97af09fa3d45004a4122c0b456f76cb59ce8f427d97a3ae28350c92ffab7f WatchSource:0}: Error finding container 32d97af09fa3d45004a4122c0b456f76cb59ce8f427d97a3ae28350c92ffab7f: Status 404 returned error can't find the container with id 32d97af09fa3d45004a4122c0b456f76cb59ce8f427d97a3ae28350c92ffab7f
Oct 31 14:22:52.208023 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:52.207968    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-8dqzr_a7fd84c4-0953-473c-a50e-950ffe2febfb/prometheus-operator/0.log"
Oct 31 14:22:52.228832 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:52.228796    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-8dqzr_a7fd84c4-0953-473c-a50e-950ffe2febfb/kube-rbac-proxy/0.log"
Oct 31 14:22:52.681894 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:52.681679    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr" event={"ID":"3e65b8e1-fa7d-4bef-bb1b-6e65ad268075","Type":"ContainerStarted","Data":"9741c2424e325034ebf96d0630f7e2a298032c1846702dd110bd0939958a5d28"}
Oct 31 14:22:52.681894 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:52.681749    4428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr" event={"ID":"3e65b8e1-fa7d-4bef-bb1b-6e65ad268075","Type":"ContainerStarted","Data":"32d97af09fa3d45004a4122c0b456f76cb59ce8f427d97a3ae28350c92ffab7f"}
Oct 31 14:22:52.682433 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:52.682028    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:22:52.707895 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:52.706951    4428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr" podStartSLOduration=1.7069278909999999 podStartE2EDuration="1.706927891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:22:52.701056556 +0000 UTC m=+4770.099405181" watchObservedRunningTime="2026-10-31 14:22:52.706927891 +0000 UTC m=+4770.105276517"
Oct 31 14:22:56.900318 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:56.900291    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kstrj_cb046f36-d918-4cdd-b2be-f2eec97ae74e/dns/0.log"
Oct 31 14:22:56.925035 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:56.925008    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kstrj_cb046f36-d918-4cdd-b2be-f2eec97ae74e/kube-rbac-proxy/0.log"
Oct 31 14:22:57.078892 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:57.077390    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z65rw_62aae01d-fd1d-43c3-9ad7-965bc6d595e2/dns-node-resolver/0.log"
Oct 31 14:22:57.733496 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:57.733462    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-68f55d596d-9ppzt_22cf051d-2e64-45a4-bb98-074e25a7769c/registry/0.log"
Oct 31 14:22:57.824667 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:57.824634    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nmwhw_2f97d160-4ad4-4546-9310-4a20d1309648/node-ca/0.log"
Oct 31 14:22:58.708128 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:22:58.707754    4428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-59nhr"
Oct 31 14:23:02.435574 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:23:02.435543    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7485898565-4fxqz_10d7c0b8-2c3c-4ead-938b-0062041d31ae/kube-auth-proxy/0.log"
Oct 31 14:23:02.580787 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:23:02.580755    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-66bvk_ca1c7578-b050-429c-8a3d-6d771125649c/istio-proxy/0.log"
Oct 31 14:23:02.652540 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:23:02.652504    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-7sz9s_aa86ed88-56ab-4fc4-b6a4-bc7732a8db25/istio-proxy/0.log"
Oct 31 14:23:02.760410 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:23:02.760381    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-n56bm_aaa34a48-48ac-4d05-8d9d-0d0c0a870f88/istio-proxy/0.log"
Oct 31 14:23:02.899768 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:23:02.899727    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-km7m9_db129029-9467-4431-bc52-40c0524ff214/payload-pre-processing/0.log"
Oct 31 14:23:02.935153 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:23:02.935127    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w9wjk_e328741d-31fd-40db-8cbb-8c9b518f4cde/payload-processing/0.log"
Oct 31 14:23:05.630526 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:23:05.630498    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jnzs8_575eac51-2cde-4d41-9de7-f6987fb83dfe/serve-healthcheck-canary/0.log"
Oct 31 14:23:05.989976 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:23:05.989949    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7485898565-4fxqz_10d7c0b8-2c3c-4ead-938b-0062041d31ae/kube-auth-proxy/0.log"
Oct 31 14:23:06.074530 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:23:06.074499    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-66bvk_ca1c7578-b050-429c-8a3d-6d771125649c/istio-proxy/0.log"
Oct 31 14:23:06.114542 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:23:06.114514    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-7sz9s_aa86ed88-56ab-4fc4-b6a4-bc7732a8db25/istio-proxy/0.log"
Oct 31 14:23:06.171553 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:23:06.171524    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-n56bm_aaa34a48-48ac-4d05-8d9d-0d0c0a870f88/istio-proxy/0.log"
Oct 31 14:23:06.239958 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:23:06.239927    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-km7m9_db129029-9467-4431-bc52-40c0524ff214/payload-pre-processing/0.log"
Oct 31 14:23:06.256431 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:23:06.256352    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w9wjk_e328741d-31fd-40db-8cbb-8c9b518f4cde/payload-processing/0.log"
Oct 31 14:23:06.259147 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:23:06.259121    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f2kwh_852969d3-8684-454c-ac42-56f80877702a/kube-rbac-proxy/0.log"
Oct 31 14:23:06.289434 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:23:06.289390    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f2kwh_852969d3-8684-454c-ac42-56f80877702a/exporter/0.log"
Oct 31 14:23:06.327131 ip-10-0-136-168 kubenswrapper[4428]: I1031 14:23:06.327074    4428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f2kwh_852969d3-8684-454c-ac42-56f80877702a/extractor/0.log"